KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #typescript #codex #claude-code #mcp #rust #deepseek-harness #ai-agent #cli

LinkedIn-Job-Scraper

★ 195 Open GitHub ↗

LinkedIn scraper to retrieve and store a live stream of job postings

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LinkedIn Job Scraper

**Program to scrape and store a constant stream of LinkedIn job postings and dozens of their respective attributes**

**Download the polished dataset and view insights at - https://www.kaggle.com/datasets/arshkon/linkedin-job-postings**

## User Configurations

### Required
- **```logins.csv```**
  - Populate with multiple LinkedIn logins
  - Specify the purpose of the login (search or detail retreiever)
  - I recommend 1-3 logins for search and the remaining for more expensive attribute retrieval
## Optional
- **```details_retriever.py```**
  - **MAX_UPDATES**: - Number of job postings to look up before sleeping. Increase with more accounts/proxies (default = 25)
  - **SLEEP_TIME**: - Seconds to sleep between every iteration (default = 60)

## Running

This program consists of 2 main scripts, running in parallel.

```python search_retriever.py``` - discovers new job postings and insert the most recent IDs and minimal attributes into the database

```python details_retriever.py``` - populates tables with complete job attributes

It's important to note that while ```search_retriever.py``` typically runs smoothly, even through your personal IP and a singular account, ```details_retriever.py``` can be a bit finicky. Each search generates approximately 25-50 results, all of which must be individually queried to obtain their attributes. To enhance its performance, I recommend the following strategies:

- Utilize multiple proxies and accounts when running details_retriever.py.
- Experiment with different time delays to find the optimal settings.
- Run details_retriever.py during periods of lower online activity, such as late-night hours and weekends, to catch up with the progress of search_retriever.py. This will ensure that both processes remain synchronized and up to date.

## Converting Database to CSV

```python to_csv.py --folder <destination folder> --database <linkedin_jobs.db>```

Creates a CSV file for each database, along with minimal preprocessing

## Database Structure

[You can find the structure of the database here](DatabaseStructure.md)

LinkedIn-Job-Scraper

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README