countly-server

# CLAUDE.md - Countly Server

This file provides guidance for Claude (Anthropic) when working with this codebase.

## Project Overview

Countly is a product analytics platform. Backend: Node.js 22+, MongoDB. Frontend: Vue 2, Element UI, Backbone (legacy). Architecture is plugin-based.

## Quick Commands

```bash
npm run start:all:dev        # Start all services (API:3001, Frontend:6001)
npx grunt dist-all           # Build static assets (required after JS changes)
npx grunt locales            # Build locale files
npx grunt sass               # Compile SASS only

# Testing
npm run test:unit            # Unit tests
npm run test:plugin -- name  # Single plugin tests
countly plugin lint name     # Lint plugin
countly shellcheck           # Validate shell scripts
```

## Critical Security Rules

**ALWAYS follow these - no exceptions:**

1. **API endpoints must use validation**:
   ```javascript
   const { validateRead, validateCreate, validateUpdate, validateDelete } = require('../../../api/utils/rights.js');
   validateRead(params, FEATURE_NAME, () => { /* handler */ });
   ```

2. **Write operations must include app_id**:
   ```javascript
   // CORRECT - prevents cross-app access
   db.collection("items").deleteOne({_id: id, app_id: params.app_id + ""});
   ```

3. **Cast user input to strings for auth**:
   ```javascript
   params.username = params.username + "";
   ```

4. **Use spawn, not exec for shell commands**:
   ```javascript
   cp.spawn("command", [userInput]);  // Safe
   // exec("command " + userInput);   // VULNERABLE
   ```

5. **Never use v-html with user data** in Vue templates.

6. **Validate user-supplied Mongo queries — reject, never strip**. Any query/filter that comes from a request and reaches `find`/`aggregate`/`update`/`delete` must be checked with the `common` helpers at the endpoint where it is first parsed (NOT inside deep helpers or the `/drill/preprocess_query` hook). The query is run exactly as submitted or the request is rejected with `400` — it is never modified.
   ```javascript
   // raw query STRING from a request param → parse + validate in one step
   var parsed = common.parseUserQuery(params.qstring.query); // accepts string OR object
   if (parsed.error) {
       log.d("Rejected user query" + common.reqInfo(params) + ": " + parsed.error);
       return common.returnMessage(params, 400, parsed.error);
   }
   var query = parsed.query; // safe to run as-is

   // ALREADY-parsed object — e.g. dbviewer parses with EJSON, or the query is
   // nested in a larger saved payload. Validate that parsed object directly:
   var parsedQuery = EJSON.parse(params.qstring.filter); // example: already parsed (EJSON / stored doc)
   var badOp = common.findUnsafeMongoOperator(parsedQuery);
   if (badOp) {
       log.d("Rejected user query" + common.reqInfo(params) + ": Query contains disallowed operator: " + badOp);
       return common.returnMessage(params, 400, "Query contains disallowed operator: " + badOp);
   }
   ```
   `$expr` is allowed; `$where`/`$function`/`$accumulator` are rejected at any depth (including nested inside `$expr`). Log the rejection at the call site using the file's `log` and `common.reqInfo(params)` (which adds the endpoint path/method, without the api_key). Do NOT pass `params` into `parseUserQuery` and do NOT log inside it.

## File Locations

| What | Where |
|------|-------|
| Plugin code | `plugins/<name>/api/api.js` |
| Vue views | `plugins/<name>/frontend/public/javascripts/countly.views.js` |
| Templates | `plugins/<name>/frontend/public/templates/` |
| Localization | `plugins/<name>/frontend/public/localization/<name>.properties` |
| Tests | `plugins/<name>/tests.js` |

## Creating API Endpoints

```javascript
var plugins = require('../../pluginManager.js');
var common = require('../../../api/utils/common.js');
const { validateRead } = require('../../../api/utils/rights.js');

const FEATURE_NAME = 'myfeature';

plugins.register("/o/myfeature", function(ob) {
    var params = ob.params;
    validateRead(params, FEATURE_NAME, function() {
        // Validate input
        var argProps = {
            'id': { 'required': true, 'type': 'String' }
        };
        var validation = common.validateArgs(params.qstring, argProps, true);
        if (!validation.obj) {
            common.returnMessage(params, 400, 'Error: ' + validation.errors);
            return;
        }
        
        // Query with app_id
        common.db.collection('mydata').findOne(
            {_id: validation.obj.id, app_id: params.app_id + ""},
            function(err, result) {
                common.returnOutput(params, result || {});
            }
        );
    });
});
```

## Creating Vue Components

```javascript
var MyComponent = countlyVue.views.create({
    template: countlyVue.T("/myplugin/templates/main.html"),
    mixins: [countlyVue.mixins.auth(FEATURE_NAME)],
    data: function() {
        return { items: [] };
    },
    computed: {
        // Prefer computed over watchers
    },
    methods: {
        refresh: function() {
            // Called for auto-refresh
        }
    }
});

app.route('/dashboard/myfeature', 'myfeature', function() {
    new countlyVue.views.BackboneWrapper({ component: MyComponent }).render();
});
```

## MongoDB Patterns

```javascript
// Read batcher for hot documents
common.readBatcher.getOne("collection", {_id: id}, callback);

// Write batcher for multiple updates
common.writeBatcher.add("collection", id, {$inc: {count: 1}});

// Always use projection
db.collection('x').findOne({_id: id}, {projection: {field: 1}});
```

## Plugin Lifecycle Hooks

```javascript
// Required if your plugin creates per-app collections
plugins.register("/i/apps/create", function(ob) {
    common.db.collection('mydata' + ob.appId).createIndex({"field": 1});
});

plugins.register("/i/apps/delete", function(ob) {
    common.db.collection('mydata' + ob.appId).drop();
});

plugins.register("/i/app_users/delete", function(ob) {
    common.db.collection("mydata" + ob.app_id).deleteMany({uid: {$in: ob.uids}});
});
```

## CSS/Styling

- Use SASS with SCSS syntax
- BEM naming with `cly-vue-` prefix
- Bulma classes use `bu-` prefix
- Use `@use`, never `@import`

## Common Anti-Patterns to Avoid

| Don't | Do Instead |
|-------|------------|
| `this.$parent.value = x` | Emit events: `this.$emit('update', x)` |
| Deep watchers | Watch specific properties |
| `v-html` with user data | Use `{{ }}` text interpolation |
| Query without app_id | Always include `app_id` in queries |
| `exec(cmd + userInput)` | `spawn(cmd, [userInput])` |
| `replace(' ', '-')` | `replace(/ /g, '-')` for all occurrences |

## Detailed Documentation

For comprehensive guidelines, read:
- `CODING_GUIDELINES.md` - Full development standards
- `docs/SECURITY.md` - Security requirements
- `docs/VUEJS_GUIDELINES.md` - Vue patterns
- `docs/CSS_STYLE_GUIDE.md` - SASS/BEM conventions
- `docs/UI_TESTING.md` - Cypress testing
- `test/README.md` - Test suite documentation
- `plugins/empty/` - Sample plugin structure

# Countly Server - AI Coding Agent Instructions

## Project Overview
Countly is a product analytics platform built with **Node.js 22+**, **MongoDB**, and **Vue 2** (with Element UI). The architecture is plugin-based: core functionality lives in `api/` and `frontend/`, while features are implemented as plugins in `plugins/`.

## Architecture

### Multi-Process Architecture
Countly runs as multiple services (start via `npm run start:all:dev`):
- **API Server** (`api/api.js`) - SDK data ingestion on port 3001
- **Frontend** (`frontend/express/app.js`) - Dashboard on port 6001
- **Job Server** (`jobServer/index.js`) - Background job processing
- **Aggregator** (`api/aggregator.js`) - Data aggregation
- **Ingestor** (`api/ingestor.js`) - High-volume data ingestion

### Plugin System
Plugins extend Countly via event hooks. Each plugin has this structure:
```
plugins/<name>/
├── api/api.js          # Backend API endpoints (required)
├── frontend/app.js     # Express middleware/routes
├── frontend/public/    # Static assets (JS, CSS, templates)
├── package.json        # Plugin metadata
├── install.js          # Installation hook
└── tests.js            # Plugin tests
```

## Backend Development Checklist

### API Endpoint Security (REQUIRED)
**Every endpoint must use validation** from `api/utils/rights.js`:
```javascript
const { validateRead, validateCreate, validateUpdate, validateDelete } = require('../../../api/utils/rights.js');

// Read operations
validateRead(params, FEATURE_NAME, () => { /* handler */ });

// Write operations - always include app_id in queries!
validateDelete(params, FEATURE_NAME, () => {
    // CORRECT: Include app_id to prevent cross-app access
    db.collection("items").deleteOne({_id: params.qstring.id, app_id: params.app_id + ""});
});
```

### Parameter Validation
Always validate and type-check input parameters:
```javascript
var argProps = {
    'name': { 'required': true, 'type': 'String' },
    'count': { 'required': false, 'type': 'Number' }
};
var validation = common.validateArgs(params.qstring.args, argProps, true);
if (!validation.obj) {
    common.returnMessage(params, 400, 'Error: ' + validation.errors);
    return false;
}

// Parse JSON safely
if (typeof params.qstring.data === "string") {
    try {
        params.qstring.data = JSON.parse(params.qstring.data);
    } catch (ex) {
        params.qstring.data = {};
    }
}
```

### MongoDB Performance
```javascript
// Use read batcher for frequently accessed documents
common.readBatcher.getOne("events", {'_id': params.app_id}, (err, event) => {});

// Use write batcher for multiple updates to same document
common.writeBatcher.add("users", id, {'$inc': updateData});

// Always use projection to limit returned fields
db.collection('plugins').findOne({_id: 'plugins'}, {projection: {'myfield': 1}});
```

### App Lifecycle Events (Required for plugins with collections)
```javascript
// Create indexes when new app is created
plugins.register("/i/apps/create", function(ob) {
    common.db.collection('app_mydata' + ob.appId).ensureIndex({"field": 1}, {background: true});
});

// Clean up when app is deleted
plugins.register("/i/apps/delete", function(ob) {
    common.db.collection('app_mydata' + ob.appId).drop();
});

// Handle user data deletion (GDPR)
plugins.register("/i/app_users/delete", function(ob) {
    common.db.collection("app_mydata" + ob.app_id).remove({uid: {$in: ob.uids}});
});
```

### Audit Logging
Log all create/update/delete actions:
```javascript
plugins.dispatch("/systemlogs", {params: params, action: "item_created", data: newItem});
plugins.dispatch("/systemlogs", {params: params, action: "item_edited", data: {before: oldItem, update: changes}});
```

## Frontend Development (Vue 2)

### Component Conventions
```javascript
// Use PascalCase for component names
var MyComponent = countlyVue.views.create({
    template: countlyVue.T("/myplugin/templates/mytemplate.html"),
    mixins: [countlyVue.mixins.auth(FEATURE_NAME)],
    data: function() { return { /* state */ }; },
    computed: { /* prefer computed over watchers */ },
    methods: { /* handlers */ }
});

// Register route with kebab-case component names in templates
app.route('/dashboard/myfeature', 'myfeature', function() {
    new countlyVue.views.BackboneWrapper({ component: MyComponent }).render();
});
```

### Vue Best Practices
- **DO**: Use `@event` instead of `v-on:event`, `:prop` instead of `v-bind:prop`
- **DO**: Prefer computed properties over data + watchers
- **DO**: Add `data-test-id` attributes for testable elements
- **DON'T**: Use `v-html` with user input (XSS risk)
- **DON'T**: Modify parent state directly - use props down, events up
- **DON'T**: Use global component registration unless truly global

### Data Test IDs for UI Testing
```html
<!-- Add data-test-id for Cypress tests -->
<button data-test-id="submit-form-button">Submit</button>
<input data-test-id="username-input" type="text">

<!-- Dynamic test IDs in Vue -->
<el-tab :data-test-id="'tab-' + tab.name + '-link'">
```

## Security Requirements

### XSS Prevention
- API output is auto-escaped via `common.returnOutput()` and `common.returnMessage()`
- Frontend: Treat API data as HTML, use `{{ msg }}` for unescaped user input
- Use `countlyCommon.encodeHtml()` for manual sanitization

### MongoDB Injection Prevention
```javascript
// Always cast credentials to strings
params.username = params.username + "";
params.password = params.password + "";
```

### File Upload Security
```javascript
// Validate file types
if (type !== "image/png" && type !== "image/gif" && type !== "image/jpeg") {
    fs.unlink(tmp_path, function() {});
    return;
}
// Sanitize filenames
var safeFileName = common.sanitizeFilename(params.qstring.filename);
```

### Command Line Security
```javascript
// Use spawn with array args, NOT exec with string concatenation
var cp = require('child_process');
cp.spawn("command", [userInput]);  // Safe
// exec("command " + userInput);   // UNSAFE - allows injection
```

## Testing

```bash
npm run test:unit                 # Unit tests (no Docker)
npm run test:api-core             # Core API tests
npm run test:lite-plugins         # CE plugin tests
npm run test:plugin -- <name>     # Single plugin tests

# Linting
countly plugin lint <pluginname>
countly plugin lintfix <pluginname>

# Shell script validation
countly shellcheck
```

### Plugin Test Requirements
- Test empty state, various inputs, and cleanup
- Verify app lifecycle handlers work correctly
- Include tests in `plugins/<name>/tests.js`

## CSS & Styling

- Use **SASS** (SCSS syntax) for stylesheets
- Use **BEM naming** with `cly-vue-` prefix for all new classes
- Use **Bulma** classes prefixed with `bu-` for grid/layout
- Don't use `@import`, use `@use` in SASS files
- Compile with `npx grunt sass` or `npx grunt dist-all`

## JSDoc Comments

Document all public functions:
```javascript
/**
 * Calculates percent change between periods.
 * @param {number} previous - data for previous period
 * @param {number} current - data for current period
 * @returns {object} {"percent": "20%", "trend": "u"}
 */
```

## Custom Scripts

Scripts in `bin/scripts/` must include:
- Header comment with description, server type, path, command
- All configurable variables with comments
- Dry run option for destructive operations
- Idempotent behavior (safe to run multiple times)

## Development Commands

```bash
npm run start:all:dev        # All services with hot reload
npx grunt dist-all           # Build all static assets (required after JS changes)
npx grunt locales            # Build locale files

# Plugin management
node bin/commands/scripts/plugin.js enable <name>
node bin/commands/scripts/plugin.js disable <name>
```

## Key Files Reference

| Purpose | Location |
|---------|----------|
| Plugin manager | `plugins/pluginManager.js` |
| Common utilities | `api/utils/common.js` |
| Authorization | `api/utils/rights.js` |
| Vue core | `frontend/express/public/javascripts/countly/vue/core.js` |
| Sample plugin | `plugins/empty/` |
| TypeScript types | `types/` |
| Coding guidelines | `CODING_GUIDELINES.md` |
| Vue.js guidelines | `docs/VUEJS_GUIDELINES.md` |
| CSS style guide | `docs/CSS_STYLE_GUIDE.md` |
| Security guidelines | `docs/SECURITY.md` |
| UI testing guide | `docs/UI_TESTING.md` |