Ciphey Documentation

File: README.md Copy Chapter

ciphey is the next generation of decoding tools, built by the same people that brought you Ciphey.

We fully intend to replace Ciphey with ciphey.

✨ You can read more about ciphey here https://skerritt.blog/introducing-ciphey/ ✨

How to Use

The simplest way to use ciphey is to join the Discord Server, head to the #bots channel and use ciphey with $ciphey. Type $help for helpful information!

The second best way is to use cargo install ciphey and call it with ciphey.

You can also git clone this repo and run docker build . it to get an image.

Features

Some features that may interest you, and that we're proud of.

Fast Copy Chapter

ciphey is fast. Very fast. Other decoders such as Ciphey require advance artifical intelligence to determine which path it should take to decode (whether to try Caesar next or Base64 etc).

ciphey is so fast we don't need to worry about this currently. For every 1 decode Ciphey can do, ciphey can do ~7. That's a 700% increase in speed.

Library First Copy Chapter

There are 2 main parts to ciphey, the library and the CLI. The CLI simply uses the library which means you can build on-top of ciphey. Some features we've built are:

• A Discord Bot
• Better testing of the whole program 💖
• This CLI

Decoders Copy Chapter

ciphey currently supports 16 decoders and it is growing fast. Ciphey supports around ~50, and we are adding more everyday.

Timer Copy Chapter

One of the big issues with Ciphey is that it could run forever. If it couldn't decode your text, you'd never know!

ciphey has a timer (built into the library and the CLI) which means it will eventually expire. The CLI defaults to 5 seconds, the Discord Bot defaults to 10 (to account for network messages being sent across).

Better Docs, Better Tests Copy Chapter

ciphey already has ~120 tests, documentation tests (to ensure our docs are kept up to date) and we enforce documentation on all of our major components. This is beautiful.

LemmeKnow Copy Chapter

LemmeKnow is the Rust version of PyWhat. It's 33 times faster which means we can now decode and determine whether something is an IP address or whatnot 3300% faster than in Python.

Multithreading Copy Chapter

Ciphey did not support multi-threading, it was quite slow. ciphey supports it natively using Rayon, one of the fastest multi-threading libraries out there.

While we do not entirely see the effects of it with only 16 decoders (and them being quite fast), as we add more decoders (and slower ones) we'll see it won't affect the overall programs speed as much.

Multi level decodings Copy Chapter

Ciphey did not support multi-level decryptions like a path of Rot13 -> Base64 -> Rot13 because it was so slow. ciphey is fast enough to support this, although we plan to turn it off eventually.

Configurable Sensitivity for Plaintext Detection Copy Chapter

ciphey now supports configurable sensitivity levels for gibberish detection, allowing for more accurate plaintext identification across different types of encodings. Classical ciphers like Caesar use Low sensitivity to better handle English-like results, while most other decoders use Medium sensitivity by default.

This feature helps reduce false positives and negatives in plaintext detection, making ciphey more reliable across a wider range of encoded texts.

Enhanced Plaintext Detection with BERT Copy Chapter

ciphey now offers enhanced plaintext detection using a BERT-based model from the gibberish-or-not crate. This feature:

• Increases plaintext detection accuracy by approximately 40%
• Reduces false positives and negatives when identifying plaintext
• Can be enabled during first-run setup or later with ciphey --enable-enhanced-detection
• Requires a one-time download of a 500MB AI model (requires a free Hugging Face account)

New Features

Better search algorithm Copy Chapter

We now use A* search. This is very fast.

A* works by using a heuristic to estimate the cost of reaching the goal from the current state.

First, we ignore the heuristic for very fast decoders like Base64 and ensure we run them first each time on each node.

Then, we calculate the heuristic for the remaining decoders using cipher_identifier which can determine the probability a given string is a certain cipher.

We store previous results in a cache to avoid recalculating the same path.

We prune the search tree to avoid unnecessary calculations and keep the memory usage down if it gets too bad.

We also keep track of statistics on decoders to dynamically prioritise decoders that work better (example: caesar is popular, but Beaufort is not so Caesar will dynamically be prioritised over Beaufort)

Finally, we keep track of popular pairs. So base64 -> base64 is very popular, so we prioritise that path (among others).

Custom themes Copy Chapter

You can now set a custom theme for ciphey. This is useful if you want to make ciphey look different.

This also helps with accessibility.

Vigenere Copy Chapter

We now use perhaps the best algorithm for Vigenere.

It's fast, accurate and handles non-letter characters better than any other algorithm.

Better English checking Copy Chapter

We use a qudgaram / trigram / english dict checker to calculate probability of plaintext.

We change the thresholds depending on the cipher. Example is that Caesar returns text that "looks" like english, whereas base64 does not.

As well as this, we have a database of popular regex (about 500) of api keys, mac addresses, etc.

We also have a is_password function to determine if a string is an exact password seen in a data dump.

More ciphers Copy Chapter

• Braille
• Atbash
• Vigenere

Database Copy Chapter

We now store statistics in a database. This is useful for seeing how ciphey is doing over time.

AI Use

We use AI for 2 things:

1. The TUI is entirely vibe coded.
2. I made AI spend hours researching every single CTF challenge out there. It created a list of 15,071 CTFs. It then went through every single CTF and looked for writeups. In those writeups it looked for anything related to encoding / decoding. It then created tests out of those. This enabled us to increase our testing coverage and make sure all CTF encoding / decoding challenges are solveable with this tool.

File: docs/changes/2024-03-11-fix-duplicate-human-prompts.md Copy Chapter

Fix Duplicate Human Verification Prompts

Issue Copy Chapter

When running ciphey in top_results mode with parallel A* search, users would sometimes see duplicate human verification prompts for the same plaintext. This occurred because:

1. The parallel A* search could discover the same solution path multiple times
2. Each discovery would trigger Athena's checker
3. The human checker would prompt for verification each time, even for identical results

Example of duplicated prompts:

🕵️ I think the plaintext is Words.
Possible plaintext: 'hello this text...' (y/N):
...
🕵️ I think the plaintext is Words.
Possible plaintext: 'hello this text...' (y/N):

Root Cause Analysis Copy Chapter

The issue stemmed from multiple factors:

1. Parallel processing in A* search allowing multiple threads to find the same solution
2. Top_results mode continuing the search after finding a valid result
3. No deduplication of human verification prompts
4. State being maintained separately in each Athena checker instance

Solution Copy Chapter

Added prompt deduplication to the human checker using a thread-safe cache:

use dashmap::DashSet;
use std::sync::OnceLock;

static SEEN_PROMPTS: OnceLock<DashSet<String>> = OnceLock::new();

fn get_seen_prompts() -> &'static DashSet<String> {
    SEEN_PROMPTS.get_or_init(|| DashSet::new())
}

The human checker now checks if it has already prompted for a given plaintext:

let prompt_key = format!("{}{}", input.description, input.text);
if !get_seen_prompts().insert(prompt_key) {
    println!("DEBUG: Skipping duplicate human verification prompt");
    return true;  // Return true to allow the search to continue
}

Benefits of this approach:

1. Thread-safe using DashSet
2. Minimal code changes required
3. Maintains existing functionality while eliminating duplicates
4. Works regardless of which code path triggered the verification

Alternative Approaches Considered Copy Chapter

1. Result deduplication in A* search - Too late, prompts already shown
2. Modifying Athena checker - More complex, required state management
3. Disabling parallel processing - Would impact performance
4. Disabling top_results mode - Would limit functionality

The chosen solution provides the best balance of:

• Minimal code changes
• No performance impact
• Preserved functionality
• Clean user experience

File: docs/changes/2024-03-21-add-vigenere-decoder.md Copy Chapter

Change: Add Vigenère Cipher Decoder

Purpose Copy Chapter

Implement a Vigenère cipher decoder to expand ciphey' classical cipher decoding capabilities. This decoder will automatically detect and break Vigenère encrypted text without requiring a key, making it valuable for cryptanalysis and historical cipher decoding.

Trade-offs Copy Chapter

Advantages Copy Section

• Implements sophisticated frequency analysis for automated key length detection
• Uses Index of Coincidence (IoC) for reliable key length determination
• Employs statistical analysis to break the cipher without requiring the key
• Handles both known-key and unknown-key scenarios

Disadvantages Copy Section

• Computationally more intensive than simple substitution ciphers
• May produce false positives with very short texts
• Effectiveness depends on text length and language characteristics

Technical Implementation Copy Chapter

• Added Vigenère decoder module with key length detection using IoC
• Implemented frequency analysis for automated key discovery
• Added comprehensive test suite with example ciphertexts
• Integrated with ciphey' existing decoder infrastructure
• Popularity score set to 0.8 reflecting its historical significance

Future Improvements Copy Chapter

• Add support for multiple languages beyond English
• Implement parallel processing for faster key space exploration
• Add option to specify known key length or partial key
• Enhance accuracy for very short ciphertexts
• Add support for variant ciphers (Beaufort, Gronsfeld)

File: docs/changes/2024-07-01-wordlist-checker.md Copy Chapter

Change: Add Wordlist Checker

Purpose Copy Chapter

Implement a wordlist checker that performs exact matching against a user-provided list of words. This allows users to check if the input text exactly matches any word in their custom wordlist, which is useful for targeted decoding tasks where the expected output is known to be within a specific set of words.

Trade-offs Copy Chapter

Advantages Copy Section

• Provides exact matching against custom wordlists
• Efficient O(1) lookups using HashSet
• Memory-mapped file handling for large wordlists (>10MB)
• Takes precedence over other checkers when specified, allowing for targeted checking
• Supports both CLI argument and config file specification

Disadvantages Copy Section

• Requires additional memory to store the wordlist
• Only performs exact matching (no partial or fuzzy matching)
• Case-sensitive matching only
• No support for multiple wordlists

Technical Implementation Copy Chapter

• Added wordlist_path and wordlist fields to the Config struct
• Implemented load_wordlist function using memory mapping for large files
• Created a new WordlistChecker that performs exact matching against the wordlist
• Updated Athena checker to prioritize wordlist checking when a wordlist is provided
• Added --wordlist CLI argument that takes precedence over config file
• Updated library API to accept pre-loaded wordlists

Future Improvements Copy Chapter

• Add support for case-insensitive matching
• Implement partial matching options
• Support multiple wordlist files
• Add progress indicator for loading large wordlists
• Implement wordlist caching
• Add support for alternative wordlist formats (CSV, JSON, etc.)

File: docs/changes/2024-07-02-sensitivity-trait.md Copy Chapter

Change: Make Sensitivity an Optional Trait

Purpose Copy Chapter

Implement an optional SensitivityAware trait for checkers that use sensitivity for gibberish detection. This separates the sensitivity functionality from the core Check trait, allowing checkers like the WordlistChecker to avoid implementing sensitivity-related methods that they don't actually use.

Trade-offs Copy Chapter

Advantages Copy Section

• Cleaner separation of concerns between core checking functionality and sensitivity handling
• Checkers that don't use sensitivity don't need to implement unused methods
• More accurate representation of which checkers actually use sensitivity
• Reduces code duplication and improves maintainability
• Makes it clearer to developers which checkers support sensitivity adjustment

Disadvantages Copy Section

• Requires changes to existing code that assumes all checkers implement sensitivity methods
• Slightly more complex trait hierarchy
• Requires careful handling in composite checkers like Athena

Technical Implementation Copy Chapter

• Created a new SensitivityAware trait in checker_type.rs with the sensitivity-related methods
• Removed sensitivity methods from the core Check trait
• Updated the WordlistChecker to not implement the SensitivityAware trait
• Updated the Athena checker to handle both sensitivity-aware and non-sensitivity-aware checkers
• Kept the sensitivity field in the Checker struct for backward compatibility
• Added documentation to clarify which checkers use sensitivity

Future Improvements Copy Chapter

• Implement the SensitivityAware trait for all checkers that actually use sensitivity
• Add runtime detection of whether a checker implements SensitivityAware
• Consider making the sensitivity field optional in the Checker struct
• Add helper methods to safely apply sensitivity only to checkers that support it
• Update documentation to clearly indicate which checkers support sensitivity adjustment

File: docs/changes/2024-07-10-astar-refactor.md Copy Chapter

Change: AStar Refactoring and String Quality Enhancement

Purpose Copy Chapter

Refactor the AStar search implementation to improve code organization and enhance string quality assessment by filtering out strings with high percentages of invisible characters.

Trade-offs Copy Chapter

Advantages Copy Section

• Improved code organization with helper functions in a separate module
• Better memory efficiency by quickly rejecting strings with >50% invisible characters
• Enhanced maintainability through clearer separation of concerns
• Easier testing of individual helper functions

Disadvantages Copy Section

• Slight increase in module complexity with an additional file
• Potential for minor performance overhead from cross-module function calls

Technical Implementation Copy Chapter

• Split AStar implementation into two files:
  • astar.rs: Core A* search algorithm implementation
  • helper_functions.rs: Supporting functions for heuristics, quality assessment, and statistics
• Enhanced calculate_string_quality function to immediately reject strings with >50% invisible characters
• Added a new test case to verify the invisible character filtering functionality
• Updated module imports and exports in mod.rs

Future Improvements Copy Chapter

• Persist decoder success statistics to disk for learning across sessions
• Further optimize string quality assessment with more sophisticated language detection
• Consider moving more common utility functions to the helper module for reuse by other search algorithms

File: docs/changes/2024-07-10-astar-simplified-heuristic.md Copy Chapter

Plan: A* Search Algorithm Rewrite for ciphey (Simplified Heuristic - Junior Engineer Edition)

Overview Copy Chapter

This document outlines the steps to rewrite the A* search algorithm in src/searchers/astar.rs for ciphey. The goal is to improve the algorithm's efficiency and focus on more promising decoding paths by:

1. Using a new node structure that explicitly defines the next decoder to use.
2. Removing the initial "do all decoders first" phase.
3. Implementing a simplified heuristic function.
4. Adding a check before node creation to prevent wasted calculations on unlikely solutions.

1. Update Node Structure (`src/searchers/astar.rs`) Copy Chapter

• Task: Modify the AStarNode struct to include the next decoder to use.

struct AStarNode {
    state: DecoderResult,
    cost: u32,
    heuristic: f32,
    total_cost: f32,
    next_decoder: Option>, // **New Field!** The next decoder to try.  Can be None for the initial node.
}

Details:

• next_decoder: This field is an Option>. It holds the specific decoder that should be applied when this node is expanded. If it's None, it means we should consider all available decoders for the next step (this is only for the initial state). The type Box<dyn Crack + Sync> is a trait object, allowing us to store any type that implements the Crack trait. The Sync bound is needed for multithreading.
• You'll need to add a use crate::decoders::interface::Crack; import to the top of the file if you don't have it already.
• Test your changes after this. Create a dummy AStarNode and ensure it compiles.

• Task: Update the DecoderResult struct in src/lib.rs to only store one decoded text (a String) and the path taken.

/// DecoderResult is the result of decoders
#[derive(Debug, Clone)]
pub struct DecoderResult {
    /// The text we have from the decoder
    pub text: String,
    /// The list of decoders we have so far
    /// The CrackResult contains more than just each decoder, such as the keys used
    /// or the checkers used.
    pub path: Vec<CrackResult>,
}

Details:

• Change text: Vec<String> to text: String. This simplifies the code and is important for this project.
• You'll have to update every part of the code that touches this struct to account for the change! Search for all usages of DecoderResult and fix the errors.
• Test your changes after this. Go through the project and fix all the compilation errors that appear as a result of these changes.

2. Remove the "Do All Decoders First" Phase (`src/searchers/astar.rs`) Copy Chapter

• Task: Remove the initial code block that runs all decoders on the input text before starting the A* search.

Details:

• This section of code is at the beginning of the astar function. It will look something like:

// First, execute all "decoder"-tagged decoders immediately
  // To ensure all simple solutions get caught
  // This is to prevent more complex decoders from returning
  // https://github.com/bee-san/ciphey/issues/79
  let mut decoder_tagged_decoders = get_decoder_tagged_decoders(&initial);

  if !decoder_tagged_decoders.components.is_empty() { ... }

• Delete this entire block of code. It's no longer needed.
• Test your changes after this. Run the existing tests to make sure you didn't accidentally break anything.

3. Implement Simplified Heuristic Function (`src/searchers/helper_functions.rs`) Copy Chapter

• Task: Modify the generate_heuristic function in src/searchers/helper_functions.rs to follow these rules (remove the hardcoded 0.3 and 0.6):

fn generate_heuristic(text: &str, path: &[CrackResult], next_decoder: &Option>) -> f32 {
    let mut base_score = 0.0;

    //1. If the decoder it wants to use is tagged "cipher" we use CipherIdentifier
    if let Some(decoder) = next_decoder {
        if decoder.get_tags().contains(&"cipher") {
            let (cipher, score) = get_cipher_identifier_score(text);
        base_score += (1.0 - (score / 100.0) as f32);
        } else {
        base_score += (1.0 - decoder.popularity);
        }
    } else {
// If next decoder is None, this is not ideal and needs to be punished.
    base_score += 0.7;
    }

//2. We do an exponential punishment based on the depth of the search tree.
//So we could do something like (0.1 * DEPTH)^2 added to the score,
//this means the deeper we are in the search tree the higher the punishment.
//Nodes really deep in the search tree tend to be crap
base_score += (0.1 * path.len() as f32).powi(2);

//3. We penalise uncommon pairings with say a 0.25 penalty.
//When people use encryption schemes they tend to do stuff like (base64 ->  base32 -> base58).
//We want to prioritise these common pairings
if path.len() > 1 {
if let Some(previous_decoder) = path.last() {
if let Some(next_decoder) = next_decoder {
if !is_common_sequence(previous_decoder.decoder, next_decoder.get_name()) {
base_score += 0.25;
}
}
}
}

    base_score
}

Details:

• Remove the multiplication by 0.6 and 0.3. This allows the base score to scale linearly with those inputs. This will mean Cipher identifier has a greater importance.
• Test your changes after this. Run the existing tests to make sure you didn't accidentally break anything.

4. Code to prevent mostly invisible/short strings from being pushed onto the stack (`src/searchers/helper_functions.rs`) Copy Chapter

• Task: Add a call to check_if_string_cant_be_decoded into astar.rs.

MyResults::Continue(results_vec) => {
                    // Process results and add to open set with heuristic prioritization
                    trace!(
                        "Processing {} results from decoders",
                        results_vec.len()
                    );

                    for r in results_vec {
                        let mut decoders_used = current_node.state.path.clone();
                        let text = r.unencrypted_text.clone();

                        // Make sure this string passes the checks
if check_if_string_cant_be_decoded(&text) {
// Add stats update for failed decoding
helper_functions::update_decoder_stats(r.decoder, false);
continue;
}

                        // Remove decoded texts from crack result
                        // Because all it holds is unencrypted text
                        let mut r = r;
                        r.unencrypted_text = None;

decoders_used.push(r);
        }
}

Details:

• We need to call this function before the node is added to the queue.

5. A* Core Logic and Node Expansion (`src/searchers/astar.rs`) Copy Chapter

• Task: Modify the main A* loop to use the new node structure.

/* Detailed source-code truncated for AI context efficiency. */

Details:

• The next_decoder field in AStarNode is now used to determine which decoder to apply next.

6. Testing (`src/searchers/astar.rs`) Copy Chapter

• Task: Update existing tests to account for the new logic.
• Task: Add new tests that specifically target the A* code path with the new logic.

Details:

• Ensure that all tests pass after the changes. If tests fail, you will need to debug the code and fix the errors.
• Write comprehensive tests to cover all possible scenarios.

File: docs/changes/2024-07-10-astar-simplified-heuristic-rewrite.md Copy Chapter

Plan: A* Search Algorithm Heuristic Rewrite for Ciphey

Overview Copy Chapter

This document outlines the plan to rewrite the heuristic function in the A* search algorithm for ciphey. The goal is to improve the algorithm's efficiency by focusing on more promising decoding paths through:

1. Using a positive heuristic for decoder popularity
2. Adding a penalty for search depth
3. Maintaining the penalty for uncommon decoder sequences
4. Removing the CipherIdentifier dependency

Current Implementation Analysis Copy Chapter

The current heuristic function in src/searchers/helper_functions.rs uses a complex system of multipliers:

• Gets a base score from the cipher identifier
• Applies penalties for uncommon sequences (1.75x multiplier)
• Applies penalties for low success rates
• Applies penalties for low popularity
• Applies penalties for low quality strings
• Applies penalties for non-printable characters

This approach has several drawbacks:

• The multipliers compound, leading to exponential growth in the heuristic value
• It's difficult to understand the contribution of each component
• It doesn't explicitly account for search depth
• It relies on CipherIdentifier, which adds complexity

Proposed Heuristic Function Copy Chapter

We'll simplify the heuristic to focus on three key components:

fn generate_heuristic(text: &str, path: &[CrackResult], next_decoder: &Option>) -> f32 {
    let mut base_score = 0.0;

    // 1. Popularity component - directly use (1.0 - popularity)
    if let Some(decoder) = next_decoder {
        base_score += (1.0 - decoder.popularity);
    } else {
        // If next decoder is None, add a moderate penalty
        base_score += 0.5;
    }

    // 2. Depth penalty - exponential growth but not too aggressive
    base_score += (0.05 * path.len() as f32).powi(2);

    // 3. Penalty for uncommon pairings
    if path.len() > 1 {
        if let Some(previous_decoder) = path.last() {
            if let Some(next_decoder) = next_decoder {
                if !is_common_sequence(previous_decoder.decoder, next_decoder.get_name()) {
                    base_score += 0.25;
                }
            }
        }
    }

    base_score
}

Component Analysis Copy Chapter

1. Popularity Component Copy Section

We'll use (1.0 - decoder.popularity) directly:

• A decoder with popularity 1.0 (most popular) would add 0.0 to the heuristic
• A decoder with popularity 0.0 (least popular) would add 1.0 to the heuristic

This makes more popular decoders more likely to be explored first.

2. Depth Penalty Copy Section

We'll use a moderate exponential function: (0.05 * path.len() as f32).powi(2)

This results in:

• Depth 5: 0.0625 penalty
• Depth 10: 0.25 penalty
• Depth 20: 1.0 penalty
• Depth 30: 2.25 penalty

This exponential growth ensures that very deep paths are significantly penalized, but it's not so aggressive that it prevents exploring moderately deep paths that might be valid solutions.

3. Uncommon Sequence Penalty Copy Section

We'll maintain the existing penalty approach, adding 0.25 to the heuristic for uncommon sequences:

• This makes common sequences relatively more attractive
• It maintains non-negative heuristic values (important for A* correctness)
• It's consistent with the rest of the heuristic design

Visual Representation Copy Chapter

graph TD
    A[Heuristic Score] --> B[Popularity Component]
    A --> C[Depth Penalty]
    A --> D[Uncommon Sequence Penalty]
    
    B --> B1[1.0 - decoder.popularity]
    B --> B2[No decoder: +0.5 penalty]
    
    C --> C1["(0.05 * depth)²"]
    
    D --> D1[Common sequence: No penalty]
    D --> D2[Uncommon sequence: +0.25]

Implementation Steps Copy Chapter

1. Update the generate_heuristic function signature to include the next_decoder parameter:

   rust Copy

   fn generate_heuristic(text: &str, path: &[CrackResult], next_decoder: &Option>) -> f32

2. Implement the simplified heuristic logic as outlined above

3. Update all calls to generate_heuristic in src/searchers/astar.rs to pass the appropriate next_decoder parameter

4. Remove any CipherIdentifier-related code that's no longer needed

5. Update tests to reflect the new heuristic behavior

Expected Benefits Copy Chapter

1. Improved Efficiency: By prioritizing more popular decoders, we'll likely find solutions faster in common cases

2. Better Path Exploration: The depth penalty will prevent the algorithm from going too deep in unproductive paths

3. Simplified Logic: The new heuristic is more straightforward and easier to understand/maintain

4. More Predictable Behavior: The additive approach makes it easier to understand how each component affects the overall heuristic

Potential Risks and Mitigations Copy Chapter

1. Risk: The depth penalty might be too aggressive or too lenient
   Mitigation: The coefficient (0.05) can be adjusted based on testing

2. Risk: Removing CipherIdentifier might reduce accuracy for certain cipher types
   Mitigation: The popularity-based approach should still work well for most cases, and we can revisit if needed

3. Risk: The penalty for uncommon sequences might be too high or too low
   Mitigation: The value (0.25) can be adjusted based on testing

Conclusion Copy Chapter

This simplified heuristic approach focuses on the most important factors for efficient A* search in the context of decoding:

• Popularity of decoders
• Depth of the search tree
• Common decoder sequences

By removing the complexity of the current implementation and focusing on these key factors, we expect to improve both the efficiency and maintainability of the A* search algorithm in ciphey.

File: docs/changes/2024-07-10-improve-string-pruning.md Copy Chapter

Change: Improve String Pruning for Low-Quality Inputs

Purpose Copy Chapter

Enhance the pruning mechanism to skip decoding of low-quality strings, which improves efficiency by avoiding wasted computation on strings that are unlikely to produce meaningful results.

Trade-offs Copy Chapter

Advantages Copy Section

• Reduces computational resources spent on strings unlikely to yield useful results
• Speeds up the overall decoding process by focusing on higher-quality candidates
• Prevents the search algorithm from exploring unproductive paths
• Improves memory usage by pruning low-quality strings early

Disadvantages Copy Section

• May occasionally reject valid encodings that have unusual characteristics
• Requires careful tuning of thresholds to balance efficiency and thoroughness
• Adds additional computation for quality checks (though this is minimal compared to the savings)

Technical Implementation Copy Chapter

• Enhanced the check_if_string_cant_be_decoded function to consider multiple quality factors:
  • String length (rejects strings with 2 or fewer characters)
  • Non-printable character ratio (rejects strings with >30% non-printable characters)
  • Overall string quality (rejects strings with quality score <0.2)
• Added comprehensive tests to verify the pruning behavior
• Updated documentation to explain the rationale behind each pruning criterion

Future Improvements Copy Chapter

• Fine-tune the thresholds based on real-world usage data
• Consider adding more sophisticated quality metrics (e.g., entropy, character distribution)
• Implement adaptive thresholds that adjust based on the search context
• Add logging to track how many strings are being pruned and why

File: docs/changes/2024-07-10-remove-cipher-mapping.md Copy Chapter

Change: Remove CIPHER_MAPPING from helper_functions

Purpose Copy Chapter

Remove the incorrect mapping between Cipher Identifier's cipher names and ciphey decoder names. The mapping was inaccurate, particularly with "fractionatedMorse" being incorrectly mapped to "morseCode" when they are different encoding schemes.

Trade-offs Copy Chapter

Advantages Copy Section

• Removes incorrect mappings that could lead to misidentification of ciphers
• Simplifies the code by directly using the first result from Cipher Identifier
• Eliminates potential confusion between different cipher types

Disadvantages Copy Section

• No longer filters cipher types based on available decoders
• May return cipher types that don't have corresponding decoders in ciphey

Technical Implementation Copy Chapter

• Removed the CIPHER_MAPPING static variable and its documentation
• Modified the get_cipher_identifier_score function to return the first result from Cipher Identifier instead of checking against the mapping
• Verified that all tests still pass after the changes

Future Improvements Copy Chapter

• Consider implementing a more accurate mapping if needed in the future
• Potentially add a check to verify if ciphey has a decoder for the identified cipher type
• Could add a more sophisticated scoring mechanism for cipher identification