cuml Agent Skills & AI Rules

# AI Code Review Guidelines - cuML C++/CUDA

**Role**: Act as a principal engineer with 10+ years experience in GPU computing and high-performance numerical computing. Focus ONLY on CRITICAL and HIGH issues.

**Target**: Sub-3% false positive rate. Be direct, concise, minimal.

**Context**: cuML C++ layer provides GPU-accelerated ML algorithm implementations using CUDA, with dependencies on RAFT, RMM, cuVS, libcudacxx, thrust, and CUB.

## IGNORE These Issues

- Style/formatting (clang-format handles this)
- Minor naming preferences (unless truly misleading)
- Personal taste on implementation (unless impacts maintainability)
- Nits that don't affect functionality
- Already-covered issues (one comment per root cause)

## CRITICAL Issues (Always Comment)

### GPU/CUDA Errors
- Unchecked CUDA errors (kernel launches, memory operations, synchronization)
- Race conditions in GPU kernels (shared memory, atomics, warps)
- Device memory leaks (cudaMalloc/cudaFree imbalance, leaked streams/events)
- Invalid memory access (out-of-bounds, use-after-free, host/device confusion)
- Missing CUDA synchronization causing non-deterministic failures
- Kernel launch with zero blocks/threads or invalid grid/block dimensions
- **Host-side integer overflow/underflow in size, launch-dim, or host-index arithmetic** (see "Integer arithmetic for sizes, launches, and host indexing" below)
- **Missing explicit stream creation for concurrent operations** (reusing default stream, missing stream isolation)
- **Incorrect stream lifecycle management** (using destroyed streams, not creating dedicated streams for concurrent ops)

### Algorithm Correctness
- Logic errors in ML algorithm kernels (clustering, regression, classification, dimensionality reduction)
- Incorrect distance metrics, kernels, or loss function implementations
- Numerical instability causing wrong results (overflow, underflow, precision loss)
- Incorrect gradient computations or convergence criteria
- **Data layout bugs** (incorrect row-major vs column-major assumptions)

### Resource Management
- GPU memory leaks (device allocations, managed memory, pinned memory)
- CUDA stream/event leaks or improper cleanup
- Missing RAII or proper cleanup. Including in exception paths.
- Resource exhaustion (GPU memory)

### API Breaking Changes
- C++ API changes without proper deprecation warnings
- Changes to data structures exposed in public headers (`cpp/include/cuml/`)
- Breaking changes to algorithm behavior

## HIGH Issues (Comment if Substantial)

### Performance Issues
- Inefficient GPU kernel launches (low occupancy, poor memory access patterns)
- Unnecessary host-device synchronization blocking GPU pipeline
- Suboptimal memory access patterns (non-coalesced, strided, unaligned)
- Excessive memory allocations in hot paths
- Warp divergence in compute-heavy kernels
- Shared memory bank conflicts

### Numerical Stability
- Floating-point operations prone to catastrophic cancellation
- Missing checks for division by zero or near-zero values
- Ill-conditioned matrix operations without preconditioning
- Accumulation errors in iterative algorithms
- Unsafe casting between numeric types (double→float with potential precision loss)
- Missing epsilon comparisons for floating-point equality checks
- **Numerical edge cases** (near-zero eigenvalues, degenerate matrices, extreme values)

### Concurrency & Thread Safety
- Race conditions in multi-GPU operations
- Improper CUDA stream management causing false dependencies
- Deadlock potential in resource acquisition
- Thread-unsafe use of global/static variables
- **Concurrent operations sharing streams incorrectly** (multi-GPU without proper isolation)
- **Stream reuse across independent operations** (causing unwanted serialization or race conditions)

### Design & Architecture
- Hard-coded GPU device IDs or resource limits
- Inappropriate use of exceptions in performance-critical paths
- Significant code duplication (3+ occurrences). Including in kernel logic.
- Reinventing functionality already available in RAFT, RMM, cuVS, libcudacxx, thrust, or CUB

### Test Quality
- Missing validation of numerical correctness
- **Using external datasets** (tests must not depend on external resources; use synthetic data or bundled datasets)

## MEDIUM Issues (Comment Selectively)

- Missing input validation (negative dimensions, null pointers)
- Deprecated CUDA API usage
- **Unclear data format in function parameters** (ambiguous row-major or column-major)

## Review Protocol

1. **CUDA correctness**: Errors checked? Memory safety? Race conditions? Synchronization?
2. **Algorithm correctness**: Does the kernel logic produce correct results? Numerical stability?
3. **Resource management**: GPU memory leaks? Stream/event cleanup?
4. **Performance**: GPU bottlenecks? Unnecessary sync? Memory access patterns?
5. **API stability**: Breaking changes to C++ APIs?
6. **Data layout**: Row/column major handled correctly?
7. **Stream lifecycle**: Are CUDA streams explicitly created/destroyed for concurrent operations?
8. **Ask, don't tell**: "Have you considered X?" not "You should do X"

## Quality Threshold

Before commenting, ask:
1. Is this actually wrong/risky, or just different?
2. Would this cause a real problem (crash, wrong results, leak)?
3. Does this comment add unique value?

**If no to any: Skip the comment.**

## Output Format

- Use severity labels: CRITICAL, HIGH, MEDIUM
- Be concise: One-line issue summary + one-line impact
- Provide code suggestions when you have concrete fixes
- No preamble or sign-off

## Examples to Follow

**CRITICAL** (GPU memory leak):
```
CRITICAL: GPU memory leak in fit()

Issue: Device memory allocated but never freed on error path
Why: Causes GPU OOM on repeated calls

Suggested fix:
if (cudaMalloc(&d_data, size) != cudaSuccess) {
    cudaFree(d_centroids);
    return ERROR_CODE;
}
```

**CRITICAL** (unchecked CUDA error):
```
CRITICAL: Unchecked kernel launch

Issue: Kernel launch error not checked
Why: Subsequent operations assume success, causing silent corruption

Suggested fix:
myKernel<<<grid, block>>>(args);
RAFT_CUDA_TRY(cudaGetLastError());
```

**HIGH** (numerical stability):
```
HIGH: Potential division by near-zero

Issue: No epsilon check before division in distance computation
Why: Can produce Inf/NaN values corrupting results
Consider: Add epsilon threshold check or use safe division helper
```

**HIGH** (performance issue):
```
HIGH: Unnecessary synchronization in hot path

Issue: cudaDeviceSynchronize() inside iteration loop
Why: Blocks GPU pipeline, 10x slowdown on benchmarks
Consider: Move sync outside loop or use streams with events
```

**CRITICAL** (data layout mismatch):
```
CRITICAL: Incorrect memory layout assumption in kernel

Issue: Kernel assumes row-major data but input is column-major
Why: Memory access pattern produces wrong results
Impact: Silent data corruption

Suggested fix:
// Check and handle data layout explicitly
if (input.is_column_major()) {
    // Use column-major kernel variant
}
```

**HIGH** (missing stream isolation):
```
HIGH: Multi-GPU operation missing dedicated streams

Issue: Multi-GPU operation uses default stream without per-device streams
Why: Can cause serialization across devices, race conditions, or deadlocks

Suggested fix:
cudaStream_t per_device_stream;
cudaStreamCreate(&per_device_stream);
// Use per_device_stream for this GPU's operations
// cudaStreamDestroy(per_device_stream) in cleanup
```

## Examples to Avoid

**Boilerplate** (avoid):
- "CUDA Best Practices: Using streams improves concurrency..."
- "Memory Management: Proper cleanup of GPU resources is important..."

**Subjective style** (ignore):
- "Consider using auto here instead of explicit type"
- "This function could be split into smaller functions"

---

## C++/CUDA-Specific Considerations

**Error Handling**:
- Use RAFT macros: `RAFT_CUDA_TRY`, `RAFT_CUBLAS_TRY`, `RAFT_CUSOLVER_TRY`
- Every CUDA call must have error checking (kernel launches, memory ops, sync)
- Use `RAFT_CUDA_TRY_NO_THROW` in destructors

**Memory Management**:
- Use RMM for device memory allocations where possible
- Use `raft::handle_t` for stream and allocator management
- Prefer RAII patterns (`rmm::device_uvector`, `rmm::device_buffer`)

**Stream Management**:
- Get streams from `raft::handle_t::get_stream()`
- For multi-stream operations, use `handle.get_internal_stream(idx)`
- Concurrent operations (multi-GPU, async ops) must have dedicated streams
- Clearly document stream lifecycle (who creates, who destroys)

**Threading**:
- Only OpenMP is allowed for host threading
- Algorithms should be thread-safe with different `raft::handle_t` instances
- Use `raft::stream_syncer` for proper stream ordering

**Public API** (`cpp/include/cuml/`):
- Functions must be stateless (POD types, `raft::handle_t`, pointers to POD)
- Doxygen documentation required for all public functions
- API changes require deprecation warnings

---

## Common Bug Patterns

### 1. Memory Layout Confusion
**Pattern**: Incorrect row-major vs column-major assumptions

**Red flags**:
- Direct pointer access without verifying data layout
- Kernel assuming row-major when data might be column-major
- Missing layout parameter in function signatures

### 2. CUDA Stream Lifecycle Issues
**Pattern**: Missing explicit stream creation for concurrent operations

**Red flags**:
- Multi-GPU operations without dedicated stream per device
- Stream creation inside loop but destruction outside loop
- Using `nullptr` or default stream for operations that need isolation
- Missing `cudaStreamDestroy` for explicitly created streams

### 3. GPU Memory Leaks
**Pattern**: Device memory allocated but not properly freed

**Red flags**:
- cudaMalloc without corresponding cudaFree
- Temporary GPU buffers allocated per iteration without cleanup
- Exception paths skipping memory cleanup
- Missing RAII or smart pointers for GPU memory

### 4. Numerical Instability in Kernels
**Pattern**: Incorrect floating-point handling in distance/kernel computations

**Red flags**:
- Division without epsilon check
- Not handling zero-norm vectors
- Accumulation without compensation (Kahan summation)
- Unsafe type casting (double→float)

### 5. Integer arithmetic for sizes, launches, and host indexing
**Pattern**: Host-side `int` (or any sub-`size_t`) arithmetic flows into an
allocation size, a kernel launch dimension, or a host pointer offset. The
product silently overflows (CWE-190) or a subtraction underflows (CWE-191)
before being widened to `size_t`, producing an undersized allocation (then GPU
heap overflow), a near-`SIZE_MAX` allocation (OOM crash), or an invalid launch
config.

**Scope**: Host code only. Device/kernel arithmetic is intentionally out of
scope — checks belong at the host-side computation site so kernels stay
branch-free.

**Required helpers** (host-side, negligible cost), declared in
`cpp/include/cuml/common/checked_arithmetic.hpp` under namespace `ML`:
- `ML::checked_mul<size_t>(a, b, ...)` — variadic, `RAFT_FAIL` on overflow.
- `ML::checked_add<size_t>(a, b, ...)` — variadic, `RAFT_FAIL` on overflow.
- `ML::checked_sub<size_t>(a, b)` — `RAFT_FAIL` on underflow.
- `ML::checked_div<size_t>(a, b)` — `RAFT_FAIL` on `b == 0` (and on signed
  `INT_MIN / -1` overflow when used with signed types).
- `ML::narrow_cast<int>(value)` — `RAFT_FAIL` if `value` does not fit in the
  target type. Use at sites where an existing API forces narrowing (e.g.
  passing a `size_t` size to a function that takes `int`, or storing
  `pair::first` into an `int`). Preserves the cast but ensures it doesn't
  silently corrupt the value. A true widening (target strictly wider than
  source) skips the magnitude check, but a negative source into an unsigned
  target still traps — sign loss is always an error.
- `ML::cuda_launch_t` — alias for the integer type expected by CUDA launch
  configuration (`dim3` components, shared-mem size). Use
  `ML::narrow_cast<ML::cuda_launch_t>(...)` for values destined for a `<<<>>>`
  grid/block dimension so the call site is self-documenting and the trap
  happens host-side instead of as a silent narrow at the launch syntax.

Use them anywhere a count-product, count-sum, count-difference, or
count-quotient is passed to an allocator, a `dim3`, a span constructor, or a
`size_t`/`int64_t` parameter. Public estimator entry points that accept
count-like `int`s should validate shape preconditions (including ordering,
e.g. `n_obs > d + s*D`) before any allocation or launch.

**Red flags**:
- `int * int` / `int + int` / `int - int` / `int / int` passed to
  `rmm::device_uvector` ctor/`.resize(...)`, `cudaMalloc*`,
  `*allocator*.allocate(...)`, `dim3(...)`, `raft::span` / `cuda::std::span`
  construction, or any `size_t` parameter.
- Subtraction of integer counts used as a length without a prior
  `RAFT_EXPECTS` / `RAFT_FAIL` guard or `checked_sub`.
- Division (including ceil-div like `(n + b - 1) / b` for launch dims) where
  the divisor is derived from a parameter and not statically known to be
  non-zero — use `checked_div` or guard explicitly.
- Cumulative offsets built by repeatedly adding `int` counts without
  `checked_add` — `offset + count` can overflow before it's used to index.
- Silent narrowing: `int n = c.size();`, `int n = m_shape.first;`,
  `foo(some_size_t)` where `foo` takes `int`, and similar — any implicit
  conversion from a wider integer type to a narrower one that involves a
  count, index, or dimension. Replace with `ML::narrow_cast<int>(...)` (or
  widen the receiver / API).
- A size guard that casts to `size_t` but the matching allocation a few lines
  down does not (the guard validates a value the allocator never sees).
- Public C/Cython entry points taking count-like `int` parameters with no
  upper-bound validation before downstream allocation.
- Grid/block dimension arithmetic in `int` where the product can plausibly
  exceed 2^31 on large inputs.
- A value computed in `int` (or `size_t`) that is passed directly into a
  `<<<grid, block, ...>>>` launch and relies on the implicit conversion to
  `unsigned int`. Require `ML::narrow_cast<ML::cuda_launch_t>(...)` so the
  conversion is checked and the call site is explicit.

---

## Code Review Checklists

### When Reviewing CUDA Kernels
- [ ] Are CUDA errors checked after kernel launch (with peek)?
- [ ] Is shared memory usage within limits and avoiding bank conflicts?
- [ ] Is shared memory used when clearly possible?
- [ ] Is thread synchronization done correctly? Are any __syncthreads call unnecessary, misplaced or missing?
- [ ] Is memory access coalesced?
- [ ] Is memory aligned?
- [ ] Is there serial work inside of a thread?
- [ ] Are warp divergence issues minimized?
- [ ] Are grid/block dimensions validated?

### When Reviewing Multi-GPU Operations
- [ ] Is stream lifecycle clearly documented?
- [ ] Are independent GPU operations using dedicated streams?
- [ ] Is `cudaSetDevice` called before device-specific operations?
- [ ] Are stream errors checked?

### When Reviewing Memory Operations
- [ ] Is data layout (row-major vs column-major) explicitly handled?
- [ ] Are device allocations paired with deallocations?
- [ ] Is RAII used for GPU resources?
- [ ] Are exception paths cleaning up resources?

### When Reviewing Numerical Computations
- [ ] Are edge cases handled (zero-norm, identical points)?
- [ ] Are divisions protected against near-zero denominators?
- [ ] Are epsilon tolerances used for floating-point comparisons?
- [ ] Is numerical stability maintained (avoiding overflow/underflow)?

### When Reviewing Tests
- [ ] Are all datasets synthetic or bundled (no external resource dependencies)?
- [ ] Is numerical correctness validated?
- [ ] Are edge cases tested (empty, single element, extreme values)?

---

**Remember**: Focus on correctness and safety. Catch real bugs (crashes, wrong results, leaks), ignore style preferences. For cuML C++: CUDA correctness and numerical stability are paramount.