Wikipedia-Scraping-with-LLM-Agents

KO | EN

gitlite — search

#javascript #python #ai #typescript #go #rust #java #nodejs #vue #pytorch #security #ios #openai #open-source

Wikipedia-Scraping-with-LLM-Agents

(★ 31)

Scraping Wikipedia by combining LangChain's agents and tools with OpenAI's LLMs and function calling

Wikipedia-Scraping-with-LLM-Agents 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# How to Scrape Wikipedia with LLM Agents
## Combining LangChain's agents and tools with OpenAI's LLMs and function calling for the web scraping of Wikipedia

Link to step-by-step guide: https://medium.com/datadriveninvestor/how-to-web-scrape-wikipedia-using-llm-agents-f0dba8400692

### Context
- The task of web scraping Wikipedia is a highly useful technique for extracting valuable information, thanks to its vast collection of structured and unstructured data. 
- Traditional tools like Selenium, while effective, tend to be manual and time-consuming.
- The impressive capabilities of large language models (LLMs) and the ability to connect them to the Internet have ushered in new possibilities in many use cases, including the domain of web scraping.
- In this article, we harness a synergistic combination of LLM agents, tools, and function calling to extract data from Wikipedia readily.

### Data
- Top 200 songs of the 2010s (from Chart2000.com - https://chart2000.com/data/chart2000-song-2010-decade-0-3-0070.csv)

### Toolkit
- LangChain
  - Agents
  - Tools
  - Output Parsers
-  OpenAI
   - LLMs (specifically `gpt-3.5-turbo-1106`)
   - Function calling

### How to
- Run `python main.py` to execute the web scraping loop for the input songs dataset

Wikipedia-Scraping-with-LLM-Agents

Wikipedia-Scraping-with-LLM-Agents 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Wikipedia-Scraping-with-LLM-Agents

Wikipedia-Scraping-with-LLM-Agents 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README