cookiecrumbler

KO | EN

gitlite — search

#javascript #python #typescript #go #android #rust #deep-learning #linux #windows #awesome #awesome-list #pytorch #ios #openai

(★ 166)

Automatically detect cookie consent notices

파일 탐색기

최종 버전 다운로드 (.zip)

- - build-deploy.yml
  - node.js.yml
- CODEOWNERS
- renovate.json
- - index.mjs
- - user-agent-override.mjs
- api.mjs
- index.html
- instrument.mjs
- lib.mjs
- puppeteer.mjs
- setup.mjs
- setupUtil.mjs
- text-classification.mjs
- util.mjs
- test.mjs
.dockerignore
.env.example
.gitignore
docker-compose.yml
docker_setup.sh
Dockerfile
eslint.config.js
LICENSE
litellm_config.yaml
package-lock.json
package.json
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# cookiecrumbler

[![License](https://img.shields.io/badge/License-MPL--2.0-blue)](https://raw.githubusercontent.com/brave/cookiecrumbler/main/LICENSE)

Cookiecrumbler automatically detects cookie consent notices on Web pages. It's intended to help with both detection of cookie consent notices that we don't currently block, and to identify webcompat reports as being related to cookie consent notice blocking.

> [!WARNING]
> This tool navigates to arbitrary URLs using a real browser and should only be run in **isolated environments**. Exposing it on a network may introduce [Server-Side Request Forgery (SSRF)](https://owasp.org/www-community/attacks/Server_Side_Request_Forgery) risks, as an attacker could use it to reach internal services or metadata endpoints. Always deploy behind appropriate network-level controls and never run it on a host with access to sensitive internal resources.

## Deployment status

Cookiecrumbler is currently being developed as a Web app which will help us run crawls and/or integrate with the webcompat reporter backend. It can also be [used as a library](using-as-a-library). In the future, we could even bundle it in the browser.

## Setup

### Local Setup
1. Install [Brave Browser](https://brave.com/linux/).

2. Install dependencies and setup browser profiles:
```bash
npm install
npm run setup -- /path/to/brave
```

Note: If browser profiles need to be updated, remove the `profiles` directory and run the setup again.

### Docker Setup
1. Ensure you have Docker and Docker Compose installed.

2. Run the initial setup:
```bash
cp .env.example .env
docker compose run --rm --entrypoint ./docker_setup.sh brave
```

Note: If browser profiles need to be updated, run the setup command again.

## Running

### Local
Start the server:
```bash
npm run serve
```

You can customize the browser and port:
```bash
npm run serve -- /usr/bin/brave-nightly 8000
```

### Docker
1. Basic setup (without LLM support):
```bash
docker compose up brave
```

2. With LLM support via Ollama:
```bash
docker compose --profile ollama up brave_ollama
```

3. With LLM support via AWS Bedrock:
```bash
aws-vault exec cookiecrumbler-bedrock -- docker compose --profile litellm up
```

> [!NOTE]
> You will need to set up the AWS profile in your environment.

For all setups, visit `localhost:3000` in your browser.

## Using as a library

```js
import { checkPage } from 'cookiecrumbler';

const result = await checkPage({
  url: 'https://example.com',        // URL to visit
  seconds: 4,                        // delay before checking for a notice
  interactive: false,                // show the browser while running?
  executablePath: '/path/to/binary', // what browser to run
  adblockLists: {                    // enable/disable filter lists by component id
    'cdbbhgbmjhfnhnmgeddbliobbofkgdhe': false,
  },
  screenshot: true,                  // return images of detected notices or full page (see Screenshots section)
  markup: true,                      // return markup of detected notices
});
```

## Screenshots

The `screenshot` parameter can be set to `true`, `false`, `always`, `fullPage`, the behavior is summarized in the following table:

| value     | element detected | no element detected |
|-----------|------------------|---------------------|
| true      | 🎯               | ❌                  |
| false     | ❌               | ❌                  |
| always    | 🎯               | 📄                  |
| fullPage  | 📄               | 📄                  |

**Legend**:  
🎯 - Screenshot of detected element  
📄 - Screenshot of full page  
❌ - No screenshot

## Testing

```
npm run test
```

You can also pass a path to a different browser binary if necessary:

```
npm run test -- /usr/bin/brave-nightly
```

### Generating test cases

There is a set of regression test cases in the `testcases` directory.

Each test case is a single self-contained HTML file.
These files can be generated using a tool like [nodeSavePageWE](https://github.com/markusmobius/nodeSavePageWE).

cookiecrumbler

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

cookiecrumbler

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README