## File: README.md

ciphey is the next generation of decoding tools, built by the same people that brought you [Ciphey](https://github.com/ciphey/ciphey).

We fully intend to replace [Ciphey](https://github.com/ciphey/ciphey) with ciphey.

✨ You can read more about ciphey here https://skerritt.blog/introducing-ciphey/ ✨

# How to Use

The simplest way to use ciphey is to join the [Discord Server](http://discord.skerritt.blog), head to the #bots channel and use ciphey with `$ciphey`. Type `$help` for helpful information!

The second best way is to use `cargo install ciphey` and call it with `ciphey`.

You can also `git clone` this repo and run `docker build .` it to get an image.

# Features

Some features that may interest you, and that we're proud of.

## Fast



ciphey is fast. Very fast. Other decoders such as Ciphey require advance artifical intelligence to determine which path it should take to decode (whether to try Caesar next or Base64 etc).

ciphey is so fast we don't need to worry about this currently. For every 1 decode Ciphey can do, ciphey can do ~7. That's a 700% increase in speed.

## Library First

There are 2 main parts to ciphey, the library and the CLI. The CLI simply uses the library which means you can build on-top of ciphey. Some features we've built are:
* [A Discord Bot](https://github.com/bee-san/discord-bot)
* Better testing of the whole program 💖
* This CLI

## Decoders

ciphey currently supports 16 decoders and it is growing [fast](https://github.com/bee-san/ciphey/issues/61). Ciphey supports around ~50, and we are adding more everyday.

## Timer

One of the big issues with Ciphey is that it could run forever. If it couldn't decode your text, you'd never know!

ciphey has a timer (built into the library and the CLI) which means it will eventually expire. The CLI defaults to 5 seconds, the Discord Bot defaults to 10 (to account for network messages being sent across).

## Better Docs, Better Tests

ciphey already has ~120 tests, documentation tests (to ensure our docs are kept up to date) and we enforce documentation on all of our major components. This is beautiful.

## LemmeKnow





[LemmeKnow](https://github.com/swanandx/lemmeknow) is the Rust version of [PyWhat](https://github.com/bee-san/pyWhat). It's 33 times faster which means we can now decode and determine whether something is an IP address or whatnot 3300% faster than in Python.

## Multithreading

Ciphey did not support multi-threading, it was quite slow. ciphey supports it natively using [Rayon](https://github.com/rayon-rs/rayon), one of the fastest multi-threading libraries out there.

While we do not entirely see the effects of it with only 16 decoders (and them being quite fast), as we add more decoders (and slower ones) we'll see it won't affect the overall programs speed as much.

## Multi level decodings

Ciphey did not support multi-level decryptions like a path of Rot13 -> Base64 -> Rot13 because it was so slow. ciphey is fast enough to support this, although we plan to turn it off eventually.

## Configurable Sensitivity for Plaintext Detection

ciphey now supports configurable sensitivity levels for gibberish detection, allowing for more accurate plaintext identification across different types of encodings. Classical ciphers like Caesar use Low sensitivity to better handle English-like results, while most other decoders use Medium sensitivity by default.

This feature helps reduce false positives and negatives in plaintext detection, making ciphey more reliable across a wider range of encoded texts.

## Enhanced Plaintext Detection with BERT

ciphey now offers enhanced plaintext detection using a BERT-based model from the `gibberish-or-not` crate. This feature:
- Increases plaintext detection accuracy by approximately 40%
- Reduces false positives and negatives when identifying plaintext
- Can be enabled during first-run setup or later with `ciphey --enable-enhanced-detection`
- Requires a one-time download of a 500MB AI model (requires a free Hugging Face account)

# New Features
## Better search algorithm
We now use A* search. This is very fast.

A* works by using a heuristic to estimate the cost of reaching the goal from the current state.

First, we ignore the heuristic for very fast decoders like Base64 and ensure we run them first each time on each node.

Then, we calculate the heuristic for the remaining decoders using `cipher_identifier` which can determine the probability a given string is a certain cipher.

We store previous results in a cache to avoid recalculating the same path.

We prune the search tree to avoid unnecessary calculations and keep the memory usage down if it gets too bad.

We also keep track of statistics on decoders to dynamically prioritise decoders that work better (example: caesar is popular, but Beaufort is not so Caesar will dynamically be prioritised over Beaufort)

Finally, we keep track of popular pairs. So base64 -> base64 is very popular, so we prioritise that path (among others).

## Custom themes

You can now set a custom theme for ciphey. This is useful if you want to make ciphey look different.

This also helps with accessibility.

## Vigenere

We now use perhaps the best algorithm for Vigenere.

It's fast, accurate and handles non-letter characters better than any other algorithm.

## Better English checking

We use a qudgaram / trigram / english dict checker to calculate probability of plaintext. 

We change the thresholds depending on the cipher. Example is that Caesar returns text that "looks" like english, whereas base64 does not.

As well as this, we have a database of popular regex (about 500) of api keys, mac addresses, etc.

We also have a `is_password` function to determine if a string is an exact password seen in a data dump.

## More ciphers
* Braille
* Atbash
* Vigenere

## Database

We now store statistics in a database. This is useful for seeing how ciphey is doing over time.

# AI Use

We use AI for 2 things:
1. The TUI is entirely vibe coded.
2. I made AI spend hours researching every single CTF challenge out there. It created a list of 15,071 CTFs. It then went through every single CTF and looked for writeups. In those writeups it looked for anything related to encoding / decoding. It then created tests out of those. This enabled us to increase our testing coverage and make sure all CTF encoding / decoding challenges are solveable with this tool.

---

## File: docs/changes/2024-03-11-fix-duplicate-human-prompts.md

# Fix Duplicate Human Verification Prompts

## Issue
When running ciphey in top_results mode with parallel A* search, users would sometimes see duplicate human verification prompts for the same plaintext. This occurred because:

1. The parallel A* search could discover the same solution path multiple times
2. Each discovery would trigger Athena's checker
3. The human checker would prompt for verification each time, even for identical results

Example of duplicated prompts:
```
🕵️ I think the plaintext is Words.
Possible plaintext: 'hello this text...' (y/N):
...
🕵️ I think the plaintext is Words.
Possible plaintext: 'hello this text...' (y/N):
```

## Root Cause Analysis
The issue stemmed from multiple factors:
1. Parallel processing in A* search allowing multiple threads to find the same solution
2. Top_results mode continuing the search after finding a valid result
3. No deduplication of human verification prompts
4. State being maintained separately in each Athena checker instance

## Solution
Added prompt deduplication to the human checker using a thread-safe cache:

```rust
use dashmap::DashSet;
use std::sync::OnceLock;

static SEEN_PROMPTS: OnceLock<DashSet<String>> = OnceLock::new();

fn get_seen_prompts() -> &'static DashSet<String> {
    SEEN_PROMPTS.get_or_init(|| DashSet::new())
}
```

The human checker now checks if it has already prompted for a given plaintext:
```rust
let prompt_key = format!("{}{}", input.description, input.text);
if !get_seen_prompts().insert(prompt_key) {
    println!("DEBUG: Skipping duplicate human verification prompt");
    return true;  // Return true to allow the search to continue
}
```

Benefits of this approach:
1. Thread-safe using DashSet
2. Minimal code changes required
3. Maintains existing functionality while eliminating duplicates
4. Works regardless of which code path triggered the verification

## Alternative Approaches Considered
1. Result deduplication in A* search - Too late, prompts already shown
2. Modifying Athena checker - More complex, required state management
3. Disabling parallel processing - Would impact performance
4. Disabling top_results mode - Would limit functionality

The chosen solution provides the best balance of:
- Minimal code changes
- No performance impact
- Preserved functionality
- Clean user experience

---

## File: docs/changes/2024-03-21-add-vigenere-decoder.md

# Change: Add Vigenère Cipher Decoder

## Purpose
Implement a Vigenère cipher decoder to expand ciphey' classical cipher decoding capabilities. This decoder will automatically detect and break Vigenère encrypted text without requiring a key, making it valuable for cryptanalysis and historical cipher decoding.

## Trade-offs
### Advantages
- Implements sophisticated frequency analysis for automated key length detection
- Uses Index of Coincidence (IoC) for reliable key length determination
- Employs statistical analysis to break the cipher without requiring the key
- Handles both known-key and unknown-key scenarios

### Disadvantages
- Computationally more intensive than simple substitution ciphers
- May produce false positives with very short texts
- Effectiveness depends on text length and language characteristics

## Technical Implementation
- Added Vigenère decoder module with key length detection using IoC
- Implemented frequency analysis for automated key discovery
- Added comprehensive test suite with example ciphertexts
- Integrated with ciphey' existing decoder infrastructure
- Popularity score set to 0.8 reflecting its historical significance

## Future Improvements
- Add support for multiple languages beyond English
- Implement parallel processing for faster key space exploration
- Add option to specify known key length or partial key
- Enhance accuracy for very short ciphertexts
- Add support for variant ciphers (Beaufort, Gronsfeld)

---

## File: docs/changes/2024-07-01-wordlist-checker.md

# Change: Add Wordlist Checker

## Purpose
Implement a wordlist checker that performs exact matching against a user-provided list of words. This allows users to check if the input text exactly matches any word in their custom wordlist, which is useful for targeted decoding tasks where the expected output is known to be within a specific set of words.

## Trade-offs

### Advantages
- Provides exact matching against custom wordlists
- Efficient O(1) lookups using HashSet
- Memory-mapped file handling for large wordlists (>10MB)
- Takes precedence over other checkers when specified, allowing for targeted checking
- Supports both CLI argument and config file specification

### Disadvantages
- Requires additional memory to store the wordlist
- Only performs exact matching (no partial or fuzzy matching)
- Case-sensitive matching only
- No support for multiple wordlists

## Technical Implementation
- Added `wordlist_path` and `wordlist` fields to the `Config` struct
- Implemented `load_wordlist` function using memory mapping for large files
- Created a new `WordlistChecker` that performs exact matching against the wordlist
- Updated Athena checker to prioritize wordlist checking when a wordlist is provided
- Added `--wordlist` CLI argument that takes precedence over config file
- Updated library API to accept pre-loaded wordlists

## Future Improvements
- Add support for case-insensitive matching
- Implement partial matching options
- Support multiple wordlist files
- Add progress indicator for loading large wordlists
- Implement wordlist caching
- Add support for alternative wordlist formats (CSV, JSON, etc.)

---

## File: docs/changes/2024-07-02-sensitivity-trait.md

# Change: Make Sensitivity an Optional Trait

## Purpose
Implement an optional `SensitivityAware` trait for checkers that use sensitivity for gibberish detection. This separates the sensitivity functionality from the core `Check` trait, allowing checkers like the WordlistChecker to avoid implementing sensitivity-related methods that they don't actually use.

## Trade-offs

### Advantages
- Cleaner separation of concerns between core checking functionality and sensitivity handling
- Checkers that don't use sensitivity don't need to implement unused methods
- More accurate representation of which checkers actually use sensitivity
- Reduces code duplication and improves maintainability
- Makes it clearer to developers which checkers support sensitivity adjustment

### Disadvantages
- Requires changes to existing code that assumes all checkers implement sensitivity methods
- Slightly more complex trait hierarchy
- Requires careful handling in composite checkers like Athena

## Technical Implementation
- Created a new `SensitivityAware` trait in `checker_type.rs` with the sensitivity-related methods
- Removed sensitivity methods from the core `Check` trait
- Updated the WordlistChecker to not implement the `SensitivityAware` trait
- Updated the Athena checker to handle both sensitivity-aware and non-sensitivity-aware checkers
- Kept the sensitivity field in the `Checker` struct for backward compatibility
- Added documentation to clarify which checkers use sensitivity

## Future Improvements
- Implement the `SensitivityAware` trait for all checkers that actually use sensitivity
- Add runtime detection of whether a checker implements `SensitivityAware`
- Consider making the sensitivity field optional in the `Checker` struct
- Add helper methods to safely apply sensitivity only to checkers that support it
- Update documentation to clearly indicate which checkers support sensitivity adjustment

---

## File: docs/changes/2024-07-10-astar-refactor.md

# Change: AStar Refactoring and String Quality Enhancement

## Purpose
Refactor the AStar search implementation to improve code organization and enhance string quality assessment by filtering out strings with high percentages of invisible characters.

## Trade-offs
### Advantages
- Improved code organization with helper functions in a separate module
- Better memory efficiency by quickly rejecting strings with >50% invisible characters
- Enhanced maintainability through clearer separation of concerns
- Easier testing of individual helper functions

### Disadvantages
- Slight increase in module complexity with an additional file
- Potential for minor performance overhead from cross-module function calls

## Technical Implementation
- Split AStar implementation into two files:
  - `astar.rs`: Core A* search algorithm implementation
  - `helper_functions.rs`: Supporting functions for heuristics, quality assessment, and statistics
- Enhanced `calculate_string_quality` function to immediately reject strings with >50% invisible characters
- Added a new test case to verify the invisible character filtering functionality
- Updated module imports and exports in `mod.rs`

## Future Improvements
- Persist decoder success statistics to disk for learning across sessions
- Further optimize string quality assessment with more sophisticated language detection
- Consider moving more common utility functions to the helper module for reuse by other search algorithms

---

## File: docs/changes/2024-07-10-astar-simplified-heuristic.md

# Plan: A* Search Algorithm Rewrite for ciphey (Simplified Heuristic - Junior Engineer Edition)

## Overview

This document outlines the steps to rewrite the A* search algorithm in `src/searchers/astar.rs` for ciphey. The goal is to improve the algorithm's efficiency and focus on more promising decoding paths by:

1.  Using a new node structure that explicitly defines the next decoder to use.
2.  Removing the initial "do all decoders first" phase.
3.  Implementing a simplified heuristic function.
4. Adding a check before node creation to prevent wasted calculations on unlikely solutions.

## 1. Update Node Structure (`src/searchers/astar.rs`)

-   **Task:** Modify the `AStarNode` struct to include the next decoder to use.

```rust
struct AStarNode {
    state: DecoderResult,
    cost: u32,
    heuristic: f32,
    total_cost: f32,
    next_decoder: Option>, // **New Field!** The next decoder to try.  Can be None for the initial node.
}
```

**Details:**

*   `next_decoder`: This field is an `Option>`. It holds the *specific* decoder that should be applied when this node is expanded. If it's `None`, it means we should consider *all* available decoders for the next step (this is only for the initial state). The type `Box<dyn Crack + Sync>` is a trait object, allowing us to store any type that implements the `Crack` trait.  The `Sync` bound is needed for multithreading.
*   You'll need to add a `use crate::decoders::interface::Crack;` import to the top of the file if you don't have it already.
*  Test your changes after this.  Create a dummy `AStarNode` and ensure it compiles.

-   **Task:** Update the `DecoderResult` struct in `src/lib.rs` to only store *one* decoded text (a `String`) and the path taken.

```rust
/// DecoderResult is the result of decoders
#[derive(Debug, Clone)]
pub struct DecoderResult {
    /// The text we have from the decoder
    pub text: String,
    /// The list of decoders we have so far
    /// The CrackResult contains more than just each decoder, such as the keys used
    /// or the checkers used.
    pub path: Vec<CrackResult>,
}
```

**Details:**

*   Change `text: Vec<String>` to `text: String`.  This simplifies the code and is important for this project.
*   You'll have to update *every* part of the code that touches this struct to account for the change! Search for all usages of `DecoderResult` and fix the errors.
*  Test your changes after this.  Go through the project and fix all the compilation errors that appear as a result of these changes.

## 2. Remove the "Do All Decoders First" Phase (`src/searchers/astar.rs`)

-   **Task:** Remove the initial code block that runs all decoders on the input text *before* starting the A* search.

**Details:**

*   This section of code is at the beginning of the `astar` function. It will look something like:
  ```rust
    // First, execute all "decoder"-tagged decoders immediately
    // To ensure all simple solutions get caught
    // This is to prevent more complex decoders from returning
    // https://github.com/bee-san/ciphey/issues/79
    let mut decoder_tagged_decoders = get_decoder_tagged_decoders(&initial);

    if !decoder_tagged_decoders.components.is_empty() { ... }
  ```
*   Delete this entire block of code. It's no longer needed.
*  Test your changes after this.  Run the existing tests to make sure you didn't accidentally break anything.

## 3. Implement Simplified Heuristic Function (`src/searchers/helper_functions.rs`)

-   **Task:** Modify the `generate_heuristic` function in `src/searchers/helper_functions.rs` to follow these rules (remove the hardcoded 0.3 and 0.6):

```rust
fn generate_heuristic(text: &str, path: &[CrackResult], next_decoder: &Option>) -> f32 {
    let mut base_score = 0.0;

    //1. If the decoder it wants to use is tagged "cipher" we use CipherIdentifier
    if let Some(decoder) = next_decoder {
        if decoder.get_tags().contains(&"cipher") {
            let (cipher, score) = get_cipher_identifier_score(text);
        base_score += (1.0 - (score / 100.0) as f32);
        } else {
        base_score += (1.0 - decoder.popularity);
        }
    } else {
// If next decoder is None, this is not ideal and needs to be punished.
    base_score += 0.7;
    }

//2. We do an exponential punishment based on the depth of the search tree.
//So we could do something like (0.1 * DEPTH)^2 added to the score,
//this means the deeper we are in the search tree the higher the punishment.
//Nodes really deep in the search tree tend to be crap
base_score += (0.1 * path.len() as f32).powi(2);

//3. We penalise uncommon pairings with say a 0.25 penalty.
//When people use encryption schemes they tend to do stuff like (base64 ->  base32 -> base58).
//We want to prioritise these common pairings
if path.len() > 1 {
if let Some(previous_decoder) = path.last() {
if let Some(next_decoder) = next_decoder {
if !is_common_sequence(previous_decoder.decoder, next_decoder.get_name()) {
base_score += 0.25;
}
}
}
}

    base_score
}
```

**Details:**

*   Remove the multiplication by `0.6` and `0.3`. This allows the base score to scale linearly with those inputs. This will mean Cipher identifier has a greater importance.
*  Test your changes after this.  Run the existing tests to make sure you didn't accidentally break anything.

## 4. Code to prevent mostly invisible/short strings from being pushed onto the stack (`src/searchers/helper_functions.rs`)

-   **Task:** Add a call to `check_if_string_cant_be_decoded` into `astar.rs`.

```rust
                MyResults::Continue(results_vec) => {
                    // Process results and add to open set with heuristic prioritization
                    trace!(
                        "Processing {} results from decoders",
                        results_vec.len()
                    );

                    for r in results_vec {
                        let mut decoders_used = current_node.state.path.clone();
                        let text = r.unencrypted_text.clone();

                        // Make sure this string passes the checks
if check_if_string_cant_be_decoded(&text) {
// Add stats update for failed decoding
helper_functions::update_decoder_stats(r.decoder, false);
continue;
}

                        // Remove decoded texts from crack result
                        // Because all it holds is unencrypted text
                        let mut r = r;
                        r.unencrypted_text = None;

decoders_used.push(r);
        }
}
```

**Details:**

*   We need to call this function *before* the node is added to the queue.

## 5. A* Core Logic and Node Expansion (`src/searchers/astar.rs`)

-   **Task:** Modify the main A* loop to use the new node structure.

```
/* Detailed source-code truncated for AI context efficiency. */
```

**Details:**

*   The `next_decoder` field in `AStarNode` is now used to determine which decoder to apply next.

## 6. Testing (`src/searchers/astar.rs`)

-   **Task:** Update existing tests to account for the new logic.
-   **Task:** Add new tests that specifically target the A* code path with the new logic.

**Details:**

*   Ensure that all tests pass after the changes. If tests fail, you will need to debug the code and fix the errors.
*   Write comprehensive tests to cover all possible scenarios.

---

## File: docs/changes/2024-07-10-astar-simplified-heuristic-rewrite.md

# Plan: A* Search Algorithm Heuristic Rewrite for Ciphey

## Overview

This document outlines the plan to rewrite the heuristic function in the A* search algorithm for ciphey. The goal is to improve the algorithm's efficiency by focusing on more promising decoding paths through:

1. Using a positive heuristic for decoder popularity
2. Adding a penalty for search depth
3. Maintaining the penalty for uncommon decoder sequences
4. Removing the CipherIdentifier dependency

## Current Implementation Analysis

The current heuristic function in `src/searchers/helper_functions.rs` uses a complex system of multipliers:
- Gets a base score from the cipher identifier
- Applies penalties for uncommon sequences (1.75x multiplier)
- Applies penalties for low success rates
- Applies penalties for low popularity
- Applies penalties for low quality strings
- Applies penalties for non-printable characters

This approach has several drawbacks:
- The multipliers compound, leading to exponential growth in the heuristic value
- It's difficult to understand the contribution of each component
- It doesn't explicitly account for search depth
- It relies on CipherIdentifier, which adds complexity

## Proposed Heuristic Function

We'll simplify the heuristic to focus on three key components:

```rust
fn generate_heuristic(text: &str, path: &[CrackResult], next_decoder: &Option>) -> f32 {
    let mut base_score = 0.0;

    // 1. Popularity component - directly use (1.0 - popularity)
    if let Some(decoder) = next_decoder {
        base_score += (1.0 - decoder.popularity);
    } else {
        // If next decoder is None, add a moderate penalty
        base_score += 0.5;
    }

    // 2. Depth penalty - exponential growth but not too aggressive
    base_score += (0.05 * path.len() as f32).powi(2);

    // 3. Penalty for uncommon pairings
    if path.len() > 1 {
        if let Some(previous_decoder) = path.last() {
            if let Some(next_decoder) = next_decoder {
                if !is_common_sequence(previous_decoder.decoder, next_decoder.get_name()) {
                    base_score += 0.25;
                }
            }
        }
    }

    base_score
}
```

## Component Analysis

### 1. Popularity Component

We'll use `(1.0 - decoder.popularity)` directly:
- A decoder with popularity 1.0 (most popular) would add 0.0 to the heuristic
- A decoder with popularity 0.0 (least popular) would add 1.0 to the heuristic

This makes more popular decoders more likely to be explored first.

### 2. Depth Penalty

We'll use a moderate exponential function: `(0.05 * path.len() as f32).powi(2)`

This results in:
- Depth 5: 0.0625 penalty
- Depth 10: 0.25 penalty
- Depth 20: 1.0 penalty
- Depth 30: 2.25 penalty

This exponential growth ensures that very deep paths are significantly penalized, but it's not so aggressive that it prevents exploring moderately deep paths that might be valid solutions.

### 3. Uncommon Sequence Penalty

We'll maintain the existing penalty approach, adding 0.25 to the heuristic for uncommon sequences:
- This makes common sequences relatively more attractive
- It maintains non-negative heuristic values (important for A* correctness)
- It's consistent with the rest of the heuristic design

## Visual Representation

```mermaid
graph TD
    A[Heuristic Score] --> B[Popularity Component]
    A --> C[Depth Penalty]
    A --> D[Uncommon Sequence Penalty]
    
    B --> B1[1.0 - decoder.popularity]
    B --> B2[No decoder: +0.5 penalty]
    
    C --> C1["(0.05 * depth)²"]
    
    D --> D1[Common sequence: No penalty]
    D --> D2[Uncommon sequence: +0.25]
```

## Implementation Steps

1. Update the `generate_heuristic` function signature to include the `next_decoder` parameter:
   ```rust
   fn generate_heuristic(text: &str, path: &[CrackResult], next_decoder: &Option>) -> f32
   ```

2. Implement the simplified heuristic logic as outlined above

3. Update all calls to `generate_heuristic` in `src/searchers/astar.rs` to pass the appropriate `next_decoder` parameter

4. Remove any CipherIdentifier-related code that's no longer needed

5. Update tests to reflect the new heuristic behavior

## Expected Benefits

1. **Improved Efficiency**: By prioritizing more popular decoders, we'll likely find solutions faster in common cases

2. **Better Path Exploration**: The depth penalty will prevent the algorithm from going too deep in unproductive paths

3. **Simplified Logic**: The new heuristic is more straightforward and easier to understand/maintain

4. **More Predictable Behavior**: The additive approach makes it easier to understand how each component affects the overall heuristic

## Potential Risks and Mitigations

1. **Risk**: The depth penalty might be too aggressive or too lenient
   **Mitigation**: The coefficient (0.05) can be adjusted based on testing

2. **Risk**: Removing CipherIdentifier might reduce accuracy for certain cipher types
   **Mitigation**: The popularity-based approach should still work well for most cases, and we can revisit if needed

3. **Risk**: The penalty for uncommon sequences might be too high or too low
   **Mitigation**: The value (0.25) can be adjusted based on testing

## Conclusion

This simplified heuristic approach focuses on the most important factors for efficient A* search in the context of decoding:
- Popularity of decoders
- Depth of the search tree
- Common decoder sequences

By removing the complexity of the current implementation and focusing on these key factors, we expect to improve both the efficiency and maintainability of the A* search algorithm in ciphey.

---

## File: docs/changes/2024-07-10-improve-string-pruning.md

# Change: Improve String Pruning for Low-Quality Inputs

## Purpose
Enhance the pruning mechanism to skip decoding of low-quality strings, which improves efficiency by avoiding wasted computation on strings that are unlikely to produce meaningful results.

## Trade-offs
### Advantages
- Reduces computational resources spent on strings unlikely to yield useful results
- Speeds up the overall decoding process by focusing on higher-quality candidates
- Prevents the search algorithm from exploring unproductive paths
- Improves memory usage by pruning low-quality strings early

### Disadvantages
- May occasionally reject valid encodings that have unusual characteristics
- Requires careful tuning of thresholds to balance efficiency and thoroughness
- Adds additional computation for quality checks (though this is minimal compared to the savings)

## Technical Implementation
- Enhanced the `check_if_string_cant_be_decoded` function to consider multiple quality factors:
  - String length (rejects strings with 2 or fewer characters)
  - Non-printable character ratio (rejects strings with >30% non-printable characters)
  - Overall string quality (rejects strings with quality score <0.2)
- Added comprehensive tests to verify the pruning behavior
- Updated documentation to explain the rationale behind each pruning criterion

## Future Improvements
- Fine-tune the thresholds based on real-world usage data
- Consider adding more sophisticated quality metrics (e.g., entropy, character distribution)
- Implement adaptive thresholds that adjust based on the search context
- Add logging to track how many strings are being pruned and why

---

## File: docs/changes/2024-07-10-remove-cipher-mapping.md

# Change: Remove CIPHER_MAPPING from helper_functions

## Purpose
Remove the incorrect mapping between Cipher Identifier's cipher names and ciphey decoder names. The mapping was inaccurate, particularly with "fractionatedMorse" being incorrectly mapped to "morseCode" when they are different encoding schemes.

## Trade-offs
### Advantages
- Removes incorrect mappings that could lead to misidentification of ciphers
- Simplifies the code by directly using the first result from Cipher Identifier
- Eliminates potential confusion between different cipher types

### Disadvantages
- No longer filters cipher types based on available decoders
- May return cipher types that don't have corresponding decoders in ciphey

## Technical Implementation
- Removed the `CIPHER_MAPPING` static variable and its documentation
- Modified the `get_cipher_identifier_score` function to return the first result from Cipher Identifier instead of checking against the mapping
- Verified that all tests still pass after the changes

## Future Improvements
- Consider implementing a more accurate mapping if needed in the future
- Potentially add a check to verify if ciphey has a decoder for the identified cipher type
- Could add a more sophisticated scoring mechanism for cipher identification