crawl4ai

(★ 78,856)

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN

File Explorer

Download Latest Version (.zip)

- - c4ai-check.md
- settings.local.json
- PR-TODOLIST.md
- - feature-requests.yml
- - bug_report.yml
  - config.yml
- - docker-release.yml
  - main.yml
  - release.yml
  - release.yml.backup
  - security.yml
  - test-release.yml.disabled
- FUNDING.yml
- pull_request_template.md
- - __init__.py
  - cli.py
- - crawler_monitor.py
- - __init__.py
- - __init__.py
  - base_strategy.py
  - bff_strategy.py
  - bfs_strategy.py
  - crazy.py
  - dfs_strategy.py
  - filters.py
  - scorers.py
- - __init__.py
  - __main__.py
  - _typing.py
  - cli.py
  - config.py
  - elements.py
  - utils.py
- - __init__.py
  - flatten_shadow_dom.js
  - navigator_overrider.js
  - remove_consent_popups.js
  - remove_overlay_elements.js
  - update_image_dimensions.js
- - __init__.py
  - cli.py
  - crawler_strategy.py
  - database.py
  - docs_manager.py
  - llmtxt.py
  - version_manager.py
  - web_crawler.py
- - __init__.py
  - c4a_compile.py
  - c4a_result.py
  - c4ai_script.py
- __init__.py
- __version__.py
- adaptive_crawler.py
- antibot_detector.py
- async_configs.py
- async_crawler_strategy.py
- async_database.py
- async_dispatcher.py
- async_logger.py
- async_url_seeder.py
- async_webcrawler.py
- browser_adapter.py
- browser_manager.py
- browser_profiler.py
- cache_context.py
- cache_validator.py
- chunking_strategy.py
- cli.py
- config.py
- content_filter_strategy.py
- content_scraping_strategy.py
- docker_client.py
- domain_mapper.py
- extraction_strategy.py
- hub.py
- install.py
- link_preview.py
- markdown_generation_strategy.py
- migrations.py
- model_loader.py
- models.py
- prompts.py
- proxy_strategy.py
- ssl_certificate.py
- table_extraction.py
- types.py
- user_agent_generator.py
- utils.py
- - .dockerignore
  - .llm.env.example
  - api.py
  - ARCHITECTURE.md
  - artifacts.py
  - auth.py
  - auth_gate.py
  - c4ai-code-context.md
  - c4ai-doc-context.md
  - config.yml
  - crawler_pool.py
  - egress_broker.py
  - egress_proxy.py
  - entrypoint.sh
  - governor.py
  - hook_registry.py
  - job.py
  - llm_broker.py
  - mcp_bridge.py
  - MIGRATION.md
  - monitor.py
  - monitor_routes.py
  - README.md
  - redis_config.py
  - requirements.txt
  - schemas.py
  - SECURITY-VERIFY.md
  - server.py
  - STRESS_TEST_PIPELINE.md
  - supervisord.conf
  - test-websocket.py
  - utils.py
  - webhook.py
  - WEBHOOK_EXAMPLES.md
  - work_queue.py
- - pitch-dark.png
  - pitch-dark.svg
  - powered-by-dark.svg
  - powered-by-disco.svg
  - powered-by-light.svg
  - powered-by-night.svg
- - release-v0.7.0.md
  - release-v0.7.1.md
  - release-v0.7.3.md
  - release-v0.7.4.md
  - release-v0.7.5.md
  - release-v0.7.6.md
  - release-v0.7.7.md
  - release-v0.7.8.md
  - release-v0.8.0.md
  - release-v0.8.5.md
  - release-v0.8.7.md
  - release-v0.8.8.md
  - release-v0.8.9.md
  - release-v0.9.0.md
  - release-v0.9.1.md
  - release-v0.9.2.md
- - browser.md
  - cli.md
- - docker-deployment.md
- - amazon_product_extraction_direct_url.py
  - amazon_product_extraction_using_hooks.py
  - amazon_product_extraction_using_use_javascript.py
  - arun_vs_arun_many.py
  - async_webcrawler_multiple_urls_example.py
  - browser_optimization_example.py
  - builtin_browser_example.py
  - chainlit.md
  - crawlai_vs_firecrawl.py
  - crawler_monitor_example.py
  - crypto_analysis_example.py
  - deep_crawl_cancellation.py
  - deep_crawl_crash_recovery.py
  - deepcrawl_example.py
  - demo_multi_config_clean.py
  - dfs_crawl_demo.py
  - dispatcher_example.py
  - docker_client_hooks_example.py
  - docker_config_obj.py
  - docker_example.py
  - docker_hooks_examples.py
  - docker_python_rest_api.py
  - docker_python_sdk.py
  - docker_webhook_example.py
  - extraction_strategies_examples.py
  - full_page_screenshot_and_pdf_export.md
  - hello_world.py
  - hello_world_undetected.py
  - hooks_example.py
  - identity_based_browsing.py
  - language_support_example.py
  - link_head_extraction_example.py
  - llm_extraction_openai_pricing.py
  - llm_markdown_generator.py
  - llm_table_extraction_example.py
  - network_console_capture_example.py
  - prefetch_two_phase_crawl.py
  - proxy_rotation_demo.py
  - quickstart.ipynb
  - quickstart.py
  - quickstart_examples_set_1.py
  - quickstart_examples_set_2.py
  - README_BUILTIN_BROWSER.md
  - regex_extraction_quickstart.py
  - research_assistant.py
  - rest_call.py
  - sample_ecommerce.html
  - scraping_strategies_performance.py
  - serp_api_project_11_feb.py
  - session_id_example.py
  - shadow_dom_crawling.py
  - simple_anti_bot_examples.py
  - ssl_example.py
  - stealth_mode_example.py
  - stealth_mode_quick_start.py
  - stealth_test_simple.py
  - storage_state_tutorial.md
  - summarize_page.py
  - table_extraction_example.py
  - tutorial_dynamic_clicks.md
  - tutorial_v0.5.py
  - undetected_simple_demo.py
  - use_geo_location.py
  - virtual_scroll_example.py
- - complete-sdk-reference.md
  - CONTRIBUTING.md
  - favicon.ico
  - index.md
  - privacy.md
  - stats.md
  - support.md
  - terms.md
- - v0.8.0-upgrade-guide.md
- - Crawl4AI_v0.3.72_Release_Announcement.ipynb
  - crawl4ai_v0_7_0_showcase.py
  - demo_v0.7.0.py
  - demo_v0.7.5.py
  - demo_v0.7.6.py
  - demo_v0.7.7.py
  - demo_v0.7.8.py
  - demo_v0.8.0.py
  - demo_v0.8.5.py
  - demo_v0.9.1.py
  - demo_v0.9.2.py
  - v0.3.74.overview.py
  - v0.7.5_docker_hooks_demo.py
  - v0.7.5_video_walkthrough.ipynb
  - v0_4_24_walkthrough.py
  - v0_4_3b2_features_demo.py
  - v0_7_0_features_demo.py
- - GHSA-DRAFT-RCE-LFI.md
- - coming_soon.md
- RELEASE_NOTES_v0.8.0.md
- prompt_net_requests.md
- README.md
- sbom.cdx.json
- gen-sbom.sh
- update_stats.py
- - compare_performance.py
  - test_adaptive_crawler.py
  - test_confidence_debug.py
  - test_embedding_performance.py
  - test_embedding_strategy.py
  - test_llm_embedding.py
  - test_query_llm_config.py
- - test_domain_mapper_adversarial.py
- - sample_wikipedia.html
  - test_0.4.2_browser_manager.py
  - test_0.4.2_config_params.py
  - test_async_doanloader.py
  - test_basic_crawling.py
  - test_browser_lifecycle.py
  - test_browser_memory.py
  - test_browser_recycle_v2.py
  - test_caching.py
  - test_chunking_and_extraction_strategies.py
  - test_content_extraction.py
  - test_content_filter_bm25.py
  - test_content_filter_prune.py
  - test_content_scraper_strategy.py
  - test_crawler_strategy.py
  - test_database_operations.py
  - test_dispatchers.py
  - test_edge_cases.py
  - test_error_handling.py
  - test_evaluation_scraping_methods_performance.configs.py
  - test_http_file_download.py
  - test_markdown_genertor.py
  - test_parameters_and_options.py
  - test_performance.py
  - test_redirect_url_resolution.py
  - test_screenshot.py
- - test_extract_pipeline.py
  - test_extract_pipeline_v2.py
- - test_browser_context_id.py
  - test_browser_manager.py
  - test_browser_manager_close.py
  - test_builtin_browser.py
  - test_builtin_strategy.py
  - test_cdp_cleanup_reuse.py
  - test_cdp_strategy.py
  - test_combined.py
  - test_context_leak_fix.py
  - test_init_script_dedup.py
  - test_launch_standalone.py
  - test_page_reuse_race_condition.py
  - test_parallel_crawling.py
  - test_playwright_strategy.py
  - test_profile_shrink.py
  - test_profiles.py
  - test_repro_1640.py
  - test_resource_filtering.py
- - __init__.py
  - conftest.py
  - test_end_to_end.py
  - test_head_fingerprint.py
  - test_real_domains.py
- - test_cli.py
- - __init__.py
  - test_deep_crawl_cancellation.py
  - test_deep_crawl_contextvar.py
  - test_deep_crawl_resume.py
  - test_deep_crawl_resume_integration.py
- - test_filter.py
- - simple_api_test.py
  - test_config_object.py
  - test_docker.py
  - test_dockerclient.py
  - test_filter_deep_crawl.py
  - test_hooks_client.py
  - test_hooks_comprehensive.py
  - test_hooks_utility.py
  - test_llm_params.py
  - test_pool_release.py
  - test_rest_api_deep_crawl.py
  - test_serialization.py
  - test_server.py
  - test_server_requests.py
  - test_server_token.py
- - generate_dummy_site.py
  - test_acyn_crawl_wuth_http_crawler_strategy.py
  - test_advanced_deep_crawl.py
  - test_async_crawler_strategy.py
  - test_async_markdown_generator.py
  - test_async_url_seeder_bm25.py
  - test_async_webcrawler.py
  - test_bff_scoring.py
  - test_cache_context.py
  - test_content_source_parameter.py
  - test_crawlers.py
  - test_deep_crawl.py
  - test_deep_crawl_filters.py
  - test_deep_crawl_scorers.py
  - test_download_file.py
  - test_flatten_shadow_dom.py
  - test_generate_schema_usage.py
  - test_http_crawler_strategy.py
  - test_llm_filter.py
  - test_max_scroll.py
  - test_mhtml.py
  - test_network_console_capture.py
  - test_persistent_context.py
  - test_robot_parser.py
  - test_schema_builder.py
  - test_stream.py
  - test_stream_dispatch.py
  - test_strip_markdown_fences.py
  - test_url_pattern.py
  - test_url_seeder_for_only_sitemap.py
  - tets_robot.py
- - test_simple.py
- - test_domain_mapper_e2e.py
- - test_logger.py
- - test_mcp_socket.py
  - test_mcp_sse.py
- - benchmark_report.py
  - cap_test.py
  - README.md
  - requirements.txt
  - run_benchmark.py
  - test_crawler_monitor.py
  - test_dispatcher_stress.py
  - test_docker_config_gen.py
  - test_stress_api.py
  - test_stress_api_xs.py
  - test_stress_docker_api.py
  - test_stress_sdk.py
- - test_create_profile.py
  - test_keyboard_handle.py
- - test_antibot_detector.py
  - test_chanel_cdp_proxy.py
  - test_persistent_proxy.py
  - test_proxy_config.py
  - test_proxy_deprecation.py
  - test_proxy_regression.py
  - test_proxy_verify.py
  - test_sticky_sessions.py
- - __init__.py
  - conftest.py
  - test_reg_browser.py
  - test_reg_config.py
  - test_reg_content.py
  - test_reg_core_crawl.py
  - test_reg_deep_crawl.py
  - test_reg_domain_mapper.py
  - test_reg_edge_cases.py
  - test_reg_extraction.py
  - test_reg_utils.py
- - test_release_0.6.4.py
  - test_release_0.7.0.py
- - test_domain_mapper_unit.py
  - test_resource_filtering_config.py
  - test_sitemap_namespace_parsing.py
- __init__.py
- check_dependencies.py
- docker_example.py
- test_arun_many.py
- test_async_logger_stderr.py
- test_bug_batch_1622_1786_1796.py
- test_cdp_changes.py
- test_cli_docs.py
- test_cloud_bugs_batch.py
- test_config_defaults.py
- test_config_matching_only.py
- test_config_selection.py
- test_docker.py
- test_docker_api_with_llm_provider.py
- test_eval_security_adversarial.py
- test_http_timeout_unit_1894.py
- test_issue_1043_mermaid_svg.py
- test_issue_1213_bm25_dedup.py
- test_issue_1370_1818_1762_1509.py
- test_issue_1484_css_selector.py
- test_issue_1594_mcp_sse.py
- test_issue_1611_llm_provider.py
- test_issue_1748_screenshot_scroll_delay.py
- test_issue_1750_screenshot_scan_full_page.py
- test_issue_1837_config_list.py
- test_issue_1842_browser_none.py
- test_issue_1848_logger_serialize.py
- test_issue_1850_mcp_sse.py
- test_link_extractor.py
- test_llm_extraction_parallel_issue_1055.py
- test_llm_simple_url.py
- test_llmtxt.py
- test_main.py
- test_markdown_generator_validation_1880.py
- test_memory_macos.py
- test_merge_head_data_scoring.py
- test_multi_config.py
- test_normalize_url.py
- test_pr_1290_1668.py
- test_pr_1435_redirected_status_code.py
- test_pr_1463_device_scale_factor.py
- test_pr_1795_1798_1734.py
- test_prefetch_integration.py
- test_prefetch_mode.py
- test_prefetch_regression.py
- test_preserve_https_for_internal_links.py
- test_pruning_preserve_whitelist_1900.py
- test_pyopenssl_security_fix.py
- test_pyopenssl_update.py
- test_raw_html_browser.py
- test_raw_html_edge_cases.py
- test_raw_html_redirected_url.py
- test_scraping_strategy.py
- test_source_sibling_selector.py
- test_table_gfm_compliance.py
- test_type_annotations.py
- test_virtual_scroll.py
- test_web_crawler.py
- test_webhook_feature.sh
- WEBHOOK_TEST_README.md
.env.txt
.gitattributes
.gitignore
CHANGELOG.md
cliff.toml
CODE_OF_CONDUCT.md
CONTRIBUTING.md
CONTRIBUTORS.md
docker-compose.yml
Dockerfile
JOURNAL.md
LICENSE
MANIFEST.in
MISSION.md
mkdocs.yml
PROGRESSIVE_CRAWLING.md
pyproject.toml
README-first.md
README.md
requirements.txt
ROADMAP.md
SECURITY-CREDITS.md
SECURITY.md
setup.cfg
setup.py
SPONSORS.md
test_llm_webhook_feature.py
test_webhook_implementation.py
uv.lock

🚀 Installation Guide

1. Get the code

git clone https://github.com/unclecode/crawl4ai

Downloads the entire project code from GitHub to your computer.

cd crawl4ai

Moves into the project folder you just downloaded.

2. Official Install Script

Easy Recommended

Prerequisites

Python 3 Python is required to use pip.

pip install -U crawl4ai

Installs the package published on PyPI directly — no need to clone the source.

pip install crawl4ai --pre

Installs the package published on PyPI directly — no need to clone the source.

pip install crawl4ai

Installs the package published on PyPI directly — no need to clone the source.

pip install crawl4ai[sync]

Installs the package published on PyPI directly — no need to clone the source.

✅ After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.

Pulled directly from this repo's README.

3. Docker

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Docker Desktop Needed to build and run containers. Install it and keep it running in the background.

docker pull unclecode/crawl4ai:latest

Type this command into your terminal and run it.

docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest

Runs the built image as an actual container.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

Pulled directly from this repo's README.

4. Python

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -U crawl4ai

Installs the package published on PyPI directly — no need to clone the source.

pip install crawl4ai --pre

Installs the package published on PyPI directly — no need to clone the source.

python -m playwright install --with-deps chromium

Runs the Python script (or module).

pip install crawl4ai

Installs the package published on PyPI directly — no need to clone the source.

python -m playwright install chromium

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

crawl4ai

File Explorer

🚀 Installation Guide

2. Official Install Script

3. Docker

4. Python

Today's picks

Was this content helpful?

Recent Feedback

crawl4ai

File Explorer

🚀 Installation Guide

2. Official Install Script

3. Docker

4. Python

Today's picks

Was this content helpful?

Recent Feedback

Download README