# Technical Documentation: joeyism/linkedin_scraper

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/joeyism/linkedin_scraper)

### Overview (`README.md`)

# LinkedIn Scraper

[](https://badge.fury.io/py/linkedin-scraper)
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/Apache-2.0)

Async LinkedIn scraper built with Playwright for extracting profile, company, and job data from LinkedIn.

## ⚠️ Breaking Changes in v3.0.0

**Version 3.0.0 introduces breaking changes and is NOT backwards compatible with previous versions.**

### What Changed:
- **Playwright instead of Selenium** - Complete rewrite using Playwright for better performance and reliability
- **Async/await throughout** - All methods are now async and require `await`
- **New package structure** - Imports have changed (e.g., `from linkedin_scraper import PersonScraper`)
- **Updated data models** - Using Pydantic models instead of simple objects
- **Different API** - Method signatures and return types have changed

### Migration Guide:

**Before (v2.x with Selenium):**
```python
from linkedin_scraper import Person

person = Person("https://linkedin.com/in/username", driver=driver)
print(person.name)
```

**After (v3.0+ with Playwright):**
```python
import asyncio
from linkedin_scraper import BrowserManager, PersonScraper

async def main():
    async with BrowserManager() as browser:
        await browser.load_session("session.json")
        scraper = PersonScraper(browser.page)
        person = await scraper.scrape("https://linkedin.com/in/username")
        print(person.name)

asyncio.run(main())
```

**If you need the old Selenium-based version:**
```bash
pip install linkedin-scraper==2.11.2
```
## Quick Testing

To test that this works, you can clone this repo, install dependencies with
```
git clone https://github.com/joeyism/linkedin_scraper.git
cd linkedin_scraper
pip3 install -e .
```
then run
```
python3 samples/create_session.py
python3 samples/scrape_company.py
python3 samples/scrape_person.py
```
and you will see the scraping in action.

---

## Features

- **Person Profiles** - Scrape comprehensive profile information
  - Basic info (name, headline, location, about)
  - Work experience with details
  - Education history
  - Skills and accomplishments
  
- **Company Pages** - Extract company information
  - Company overview and details
  - Industry and size
  - Headquarters location
  
- **Company Posts** - Scrape posts from company pages
  - Post content and text
  - Reactions, comments, reposts counts
  - Posted date and images
  
- **Job Listings** - Scrape job postings
  - Job details and requirements
  - Company information
  - Application links

- **Async/Await** - Modern async Python with Playwright
- **Type Safety** - Full Pydantic models for all data
- **Progress Callbacks** - Track scraping progress
- **Session Management** - Reuse authenticated sessions

## Installation

```bash
pip install linkedin-scraper
```

### Install Playwright browsers:

```bash
playwright install chromium
```

## Quick Start

### Basic Usage

```python
import asyncio
from linkedin_scraper import BrowserManager, PersonScraper

async def main():
    # Initialize browser
    async with BrowserManager(headless=False) as browser:
        # Load authenticated session
        await browser.load_session("session.json")
        
        # Create scraper
        scraper = PersonScraper(browser.page)
        
        # Scrape a profile
        person = await scraper.scrape("https://linkedin.com/in/williamhgates/")
        
        # Access data
        print(f"Name: {person.name}")
        print(f"Headline: {person.headline}")
        print(f"Location: {person.location}")
        print(f"Experiences: {len(person.experiences)}")
        print(f"Education: {len(person.educations)}")

asyncio.run(main())
```

### Company Scraping

```python
from linkedin_scraper import CompanyScraper

async def scrape_company():
    async with BrowserManager(headless=False) as browser:
        await browser.load_session("session.json")
        
        scraper = CompanyScraper(browser.page)
        company = await scraper.scrape("https://linkedin.com/company/microsoft/")
        
        print(f"Company: {company.name}")
        print(f"Industry: {company.industry}")
        print(f"Size: {company.company_size}")
        print(f"About: {company.about_us[:200]}...")

asyncio.run(scrape_company())
```

### Job Scraping

```python
from linkedin_scraper import JobSearchScraper

async def search_jobs():
    async with BrowserManager(headless=False) as browser:
        await browser.load_session("session.json")
        
        scraper = JobSearchScraper(browser.page)
        jobs = await scraper.search(
            keywords="Python Developer",
            location="San Francisco",
            limit=10
        )
        
        for job in jobs:
            print(f"{job.title} at {job.company}")
            print(f"Location: {job.location}")
            print(f"Link: {job.linkedin_url}")
            print("---")

asyncio.run(search_jobs())
```

### Company Posts Scraping

```python
from linkedin_scraper import BrowserManager, CompanyPostsScraper

async def scrape_company_posts():
    async with BrowserManager(headless=False) as browser:
        await browser.load_session("session.json")
        
        scraper = CompanyPostsScraper(browser.page)
        posts = await scraper.scrape(
            "https://linkedin.com/company/microsoft/",
            limit=10
        )
        
        for post in posts:
            print(f"Posted: {post.posted_date}")
            print(f"Text: {post.text[:200]}...")
            print(f"Reactions: {post.reactions_count}")
            print(f"Comments: {post.comments_count}")
            print(f"URL: {post.linkedin_url}")
            print("---")

asyncio.run(scrape_company_posts())
```

## Authentication

LinkedIn requires authentication. You need to create a session file first:

### Option 1: Manual Login Script

```python
from linkedin_scraper import BrowserManager, wait_for_manual_login

async def create_session():
    async with BrowserManager(headless=False) as browser:
        # Navigate to LinkedIn
        await browser.page.goto("https://www.linkedin.com/login")
        
        # Wait for manual login (opens browser)
        print("Please log in to LinkedIn...")
        await wait_for_manual_login(browser.page, timeout=300)
        
        # Save session
        await browser.save_session("session.json")
        print("✓ Session saved!")

asyncio.run(create_session())
```

### Option 2: Programmatic Login

```python
from linkedin_scraper import BrowserManager, login_with_credentials
import os

async def login():
    async with BrowserManager(headless=False) as browser:
        # Login with credentials
        await login_with_credentials(
            browser.page,
            username=os.getenv("LINKEDIN_EMAIL"),
            password=os.getenv("LINKEDIN_PASSWORD")
        )
        
        # Save session for reuse
        await browser.save_session("session.json")

asyncio.run(login())
```

## Progress Tracking

Track scraping progress with callbacks:

```python
from linkedin_scraper import ConsoleCallback, PersonScraper

async def scrape_with_progress():
    callback = ConsoleCallback()  # Prints progress to console
    
    async with BrowserManager(headless=False) as browser:
        await browser.load_session("session.json")
        
        scraper = PersonScraper(browser.page, callback=callback)
        person = await scraper.scrape("https://linkedin.com/in/williamhgates/")

asyncio.run(scrape_with_progress())
```

### Custom Callbacks

```python
from linkedin_scraper import ProgressCallback

class MyCallback(ProgressCallback):
    async def on_start(self, scraper_type: str, url: str):
        print(f"Starting {scraper_type} scraping: {url}")
    
    async def on_progress(self, message: str, percent: int):
        print(f"[{percent}%] {message}")
    
    async def on_complete(self, scraper_type: str, url: str):
        print(f"Completed {scraper_type}: {url}")
    
    async def on_error(self, error: Exception):
        print(f"Error: {error}")
```

## Data Models

All scraped data is returned as Pydantic models:

### Person

```python
class Person(BaseModel):
    name: str
    headline: Optional[str]
    location: Optional[str]
    about: Optional[str]
    linkedin_url: str
    experiences: List[Experience]
    educations: List[Education]
    skills: List[str]
    accomplishments: Optional[Accomplishment]
```

### Company

```python
class Company(BaseModel):
    name: str
    industry: Optional[str]
    company_size: Optional[str]
    headquarters: Optional[str]
    founded: Optional[str]
    specialties: List[str]
    about: Optional[str]
    linkedin_url: str
```

### Job

```python
class Job(BaseModel):
    title: str
    company: str
    location: Optional[str]
    description: Optional[str]
    employment_type: Optional[str]
    seniority_level: Optional[str]
    linkedin_url: str
```

### Post

```python
class Post(BaseModel):
    linkedin_url: Optional[str]
    urn: Optional[str]
    text: Optional[str]
    posted_date: Optional[str]
    reactions_count: Optional[int]
    comments_count: Optional[int]
    reposts_count: Optional[int]
    image_urls: List[str]
```

## Advanced Usage

### Browser Configuration

```python
browser = BrowserManager(
    headless=False,  # Show browser window
    slow_mo=100,     # Slow down operations (ms)
    viewport={"width": 1920, "height": 1080},
    user_agent="Custom User Agent"
)
```

### Error Handling

```python
from linkedin_scraper import (
    AuthenticationError,
    RateLimitError,
    ProfileNotFoundError
)

try:
    person = await scraper.scrape(url)
except AuthenticationError:
    print("Not logged in - session expired")
except RateLimitError:
    print("Rate limited by LinkedIn")
except ProfileNotFoundError:
    print("Profile not found or private")
```

## Best Practices

1. **Rate Limiting** - Add delays between requests
   ```python
   import asyncio
   await asyncio.sleep(2)  # 2 second delay
   ```

2. **Session Reuse** - Save and reuse sessions to avoid frequent logins

3. **Error Handling** - Always handle exceptions (rate limits, auth errors, etc.)

4. **Headless Mode** - Use `headless=False` during development, `True` for production

5. **Respect LinkedIn** - Don't scrape aggressively, respect rate limits

## Requirements

- Python 3.8+
- Playwright
- Pydantic 2.0+
- aiofiles
- python-dotenv (optional, for credentials)

## License

Apache License 2.0 - see [LICENSE](LICENSE) file for details.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Disclaimer

This tool is for educational purposes only. Make sure to comply with LinkedIn's Terms of Service and use responsibly. The authors are not responsible for any misuse of this tool.

## Links

- [GitHub Repository](https://github.com/joeyism/linkedin_scraper)
- [Issue Tracker](https://github.com/joeyism/linkedin_scraper/issues)
- [PyPI Package](https://pypi.org/project/linkedin-scraper/)

---

### Overview (`tests/README.md`)

# LinkedIn Scraper Tests

This directory contains pytest tests for the linkedin_scraper library.

## Setup

1. Install test dependencies:
```bash
pip install -r requirements.txt
```

2. For integration tests, create a LinkedIn session:
```bash
python setup_session.py
```

This will create `linkedin_session.json` which is required for integration tests.

## Running Tests

### Run all tests
```bash
pytest
```

### Run only unit tests (no LinkedIn session required)
```bash
pytest -m unit
```

### Run only integration tests (requires LinkedIn session)
```bash
pytest -m integration
```

### Run with verbose output
```bash
pytest -v
```

### Run specific test file
```bash
pytest tests/test_person_scraper.py
```

### Run specific test
```bash
pytest tests/test_person_scraper.py::test_person_scraper_basic
```

### Skip slow tests
```bash
pytest -m "not slow"
```

## Test Structure

```
tests/
├── conftest.py                  # Pytest fixtures and configuration
├── test_person_scraper.py       # PersonScraper tests
├── test_company_scraper.py      # CompanyScraper tests
├── test_job_scraper.py          # Job and JobSearch tests
├── test_browser.py              # BrowserManager tests
└── test_auth.py                 # Authentication tests
```

## Test Markers

- `@pytest.mark.unit` - Unit tests (no external dependencies)
- `@pytest.mark.integration` - Integration tests (requires LinkedIn session)
- `@pytest.mark.slow` - Slow running tests
- `@pytest.mark.asyncio` - Async tests

## Fixtures

Available fixtures (defined in `conftest.py`):

- `browser` - Provides BrowserManager instance
- `browser_with_session` - Provides BrowserManager with loaded session
- `silent_callback` - Provides SilentCallback for testing
- `test_profile_urls` - Dictionary of test LinkedIn profile URLs
- `test_company_urls` - Dictionary of test company URLs
- `test_job_search_params` - Job search parameters for testing

## Example

```python
import pytest
from linkedin_scraper import PersonScraper

@pytest.mark.integration
@pytest.mark.asyncio
async def test_my_scraper(browser_with_session, silent_callback):
    scraper = PersonScraper(browser_with_session.page, callback=silent_callback)
    person = await scraper.scrape("https://linkedin.com/in/profile")
    assert person.name is not None
```

## CI/CD Integration

For CI/CD pipelines, you can:

1. Run only unit tests (no LinkedIn session needed):
```bash
pytest -m unit
```

2. Or mock the session for integration tests (advanced).

## Important Notes

### Headless Mode
Integration tests run with `headless=False` because LinkedIn may block or behave differently in headless mode. This means:
- Browser windows will open during integration tests
- Tests will be slower than headless tests
- You'll see the browser automation in action

If you want headless tests, you need to implement stealth mode or use unit tests only.

### Test Speed
- **Unit tests**: Fast (~1-2 seconds total)
- **Integration tests**: Slow (5-30 seconds per test) due to:
  - Browser launches
  - Network requests to LinkedIn
  - Page rendering and waiting

## Troubleshooting

### "Session file not found"
Run `python setup_session.py` to create a LinkedIn session.

### Tests timing out or hanging
- Integration tests require browser windows (headless=False)
- Make sure you have Xvfb or a display available in headless environments
- Or run only unit tests: `pytest -m unit`

### "Rate limited" or no data returned
LinkedIn may rate limit requests or block scraping. Solutions:
- Add delays between tests
- Run tests individually instead of all at once
- Refresh your session by running `setup_session.py` again

### Playwright not installed
```bash
playwright install chromium
```

### Browser opens but tests fail
- Ensure your session is valid (re-run `setup_session.py`)
- Check if LinkedIn requires re-authentication
- LinkedIn's UI may have changed (selectors need updating)

---

--- METRICS ---
- Files Extracted: 2
- Estimated Token Budget: ~3744 tokens
- Canonical Reference: https://codewiki.google/github.com/joeyism/linkedin_scraper