# Deduplication Tuning (Open Source)

The Open Source edition of DefectDojo uses settings files and environment variables to tune deduplication.

See also: [Open Source Configuration](https://docs.defectdojo.com/get_started/open_source/configuration/) for details on environment variables and `local_settings.py` overrides.

## What you can configure

- **Algorithm per parser**: Choose one of Unique ID From Tool, Hash Code, Unique ID From Tool or Hash Code, or Legacy (OS only).
- **Hash fields per scanner**: Decide which fields contribute to the hash for each parser.
- **Allow null CWE**: Control whether a missing/zero CWE is acceptable when hashing.
- **Endpoint consideration**: Optionally use endpoints for deduplication when they’re not part of the hash.
- **Always-included fields**: Add fields (e.g., `service`) to all hashes regardless of per-scanner settings.

## Key settings (defaults shown)

All defaults are defined in `dojo/settings/settings.dist.py`. Override via environment or `local_settings.py`.

### Algorithm per parser

- Setting: `DEDUPLICATION_ALGORITHM_PER_PARSER`
- Values per parser: one of `unique_id_from_tool`, `hash_code`, `unique_id_from_tool_or_hash_code`, `legacy`.
- Example (env variable JSON string):

```bash
DD_DEDUPLICATION_ALGORITHM_PER_PARSER='{"Trivy Scan": "hash_code", "Veracode Scan": "unique_id_from_tool_or_hash_code"}'
```

### Hash fields per scanner

- Setting: `HASHCODE_FIELDS_PER_SCANNER`
- Example default for Trivy in OS:

```startline:endline:dojo/settings/settings.dist.py
1318:1321:dojo/settings/settings.dist.py
    "Trivy Operator Scan": ["title", "severity", "vulnerability_ids", "description"],
    "Trivy Scan": ["title", "severity", "vulnerability_ids", "cwe", "description"],
    "TFSec Scan": ["severity", "vuln_id_from_tool", "file_path", "line"],
    "Snyk Scan": ["vuln_id_from_tool", "file_path", "component_name", "component_version"],
```

- Override example (env variable JSON string):

```bash
DD_HASHCODE_FIELDS_PER_SCANNER='{"ZAP Scan":["title","cwe","severity"],"Trivy Scan":["title","severity","vulnerability_ids","description"]}'
```

### Allow null CWE per scanner

- Setting: `HASHCODE_ALLOWS_NULL_CWE`
- Controls per parser whether a null/zero CWE is acceptable in hashing. If False and the finding has `cwe = 0`, the hash falls back to the legacy computation for that finding.

### Always-included fields in hash

- Setting: `HASH_CODE_FIELDS_ALWAYS`
- Default: `[
    "service"
]`
- Impact: Appended to the hash for every scanner. Removing `service` here stops it from affecting hashes across the board.

```startline:endline:dojo/settings/settings.dist.py
1464:1466:dojo/settings/settings.dist.py
# Adding fields to the hash_code calculation regardless of the previous settings
HASH_CODE_FIELDS_ALWAYS = ["service"]
```

### Optional endpoint-based dedupe

- Setting: `DEDUPE_ALGO_ENDPOINT_FIELDS`
- Default: 
`[
    "host",
    "path"
]`
- Purpose: If endpoints are not part of the hash fields, you can still require a minimal endpoint match to deduplicate. If the list is empty `[]`, endpoints are ignored on the dedupe path.

```startline:endline:dojo/settings/settings.dist.py
1491:1499:dojo/settings/settings.dist.py
# Allows to deduplicate with endpoints if endpoints is not included in the hashcode.
# Possible values are: scheme, host, port, path, query, fragment, userinfo, and user.
# If a finding has more than one endpoint, only one endpoint pair must match to mark the finding as duplicate.
DEDUPE_ALGO_ENDPOINT_FIELDS = ["host", "path"]
```

## Endpoints: how to tune

Endpoints can affect deduplication via two mechanisms:

1. Include `endpoints` in `HASHCODE_FIELDS_PER_SCANNER` for a parser. Then endpoints are part of the hash and must match exactly according to the parser’s hashing rules.
2. If endpoints are not in the hash fields, use `DEDUPLE_ALGO_ENDPOINT_FIELDS` to specify attributes to compare. Examples:
   - `[]`: endpoints are ignored for dedupe.
   - `[