# DataGrid Formatting DSL

## Introduction

This document describes the Domain Specific Language (DSL) for defining formatting rules in the DataGrid component.

### Purpose

The DSL provides a concise, readable way to define conditional formatting rules for cells, rows, and columns. It allows users to:

- Apply styles (colors, fonts, decorations) based on cell values
- Format values for display (currencies, dates, percentages)
- Reference other cells for cross-column/row comparisons

### Two Modes

The formatting system offers two interfaces:

| Mode | Target Users | Description |
|------|--------------|-------------|
| **Advanced (DSL)** | Developers, power users | Text-based rules with Python-like syntax |
| **Basic (GUI)** | End users | Visual rule builder (80/20 coverage) |

This document focuses on the **Advanced DSL mode**, which covers 100% of formatting capabilities.

---

## Complete Syntax

### Overview

A DSL document consists of one or more **scopes**, each containing one or more **rules**:

```python
<scope>:
    <rule>
    <rule>
    ...

<scope>:
    <rule>
    ...
```

Rules are indented (Python-style) under their scope.

### Scopes

Scopes define which cells a rule applies to:

| Scope | Syntax | Applies To |
|-------|--------|------------|
| **Column** | `column <name>:` | All cells in the column |
| **Row** | `row <index>:` | All cells in the row |
| **Cell (coordinates)** | `cell (<col>, <row>):` | Single cell by position |
| **Cell (ID)** | `cell <cell_id>:` | Single cell by ID |

**Column scope:**

```python
# Simple column name
column amount:
    style("error") if value < 0

# Column name with spaces (quoted)
column "total amount":
    format("EUR")

# Both syntaxes are valid
column status:
column "status":
```

**Row scope:**

```python
# Row by index (0-based)
row 0:
    style("neutral", bold=True)

row 5:
    style("highlight")
```

**Cell scope:**

```python
# By coordinates (column, row)
cell (amount, 3):
    style("highlight")

cell ("total amount", 0):
    style("neutral", bold=True)

# By cell ID
cell tcell_grid1-3-2:
    style(background_color="yellow")
```

### Rules

A rule consists of optional **style**, optional **format**, and optional **condition**:

```
[style(...)] [format(...)] [if <condition>]
```

At least one of `style()` or `format()` must be present.

**Examples:**

```python
# Style only
style("error")

# Format only
format("EUR")

# Style + Format
style("error") format("EUR")

# With condition
style("error") if value < 0

# All combined
style("error") format("EUR") if value < 0
```

### Style

The `style()` function applies visual formatting to cells.

**Syntax:**

```python
style(<preset>)
style(<preset>, <options>)
style(<options>)
```

**Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `preset` | string (positional) | Preset name (optional) |
| `background_color` | string | Background color |
| `color` | string | Text color |
| `bold` | boolean | Bold text |
| `italic` | boolean | Italic text |
| `underline` | boolean | Underlined text |
| `strikethrough` | boolean | Strikethrough text |
| `font_size` | string | Font size (e.g., "12px", "0.9em") |

**Available presets (DaisyUI 5):**

| Preset | Background | Text |
|--------|------------|------|
| `primary` | `var(--color-primary)` | `var(--color-primary-content)` |
| `secondary` | `var(--color-secondary)` | `var(--color-secondary-content)` |
| `accent` | `var(--color-accent)` | `var(--color-accent-content)` |
| `neutral` | `var(--color-neutral)` | `var(--color-neutral-content)` |
| `info` | `var(--color-info)` | `var(--color-info-content)` |
| `success` | `var(--color-success)` | `var(--color-success-content)` |
| `warning` | `var(--color-warning)` | `var(--color-warning-content)` |
| `error` | `var(--color-error)` | `var(--color-error-content)` |

**Examples:**

```python
# Preset only
style("error")

# Preset with overrides
style("error", bold=True)
style("success", italic=True, underline=True)

# No preset, direct properties
style(color="red", bold=True)
style(background_color="#ffeeee", color="#cc0000")
```

### Format

The `format()` function transforms cell values for display.

**Syntax:**

```python
format(<preset>)
format(<preset>, <options>)
format.<type>(<options>)
```

**Using presets:**

```python
# Preset only
format("EUR")

# Preset with overrides
format("EUR", precision=3)
format("percentage", precision=0)
```

**Available presets:**

| Preset | Type | Description |
|--------|------|-------------|
| `EUR` | number | Euro currency (1 234,56 €) |
| `USD` | number | US Dollar ($1,234.56) |
| `percentage` | number | Percentage (×100, adds %) |
| `short_date` | date | DD/MM/YYYY |
| `iso_date` | date | YYYY-MM-DD |
| `yes_no` | boolean | Yes/No |

**Using explicit types:**

When not using a preset, specify the type explicitly:

```python
format.number(precision=2, suffix=" €", thousands_sep=" ")
format.date(format="%d/%m/%Y")
format.boolean(true_value="Oui", false_value="Non")
format.text(max_length=50, ellipsis="...")
format.enum(source={"draft": "Draft", "published": "Published"})
```

**Type-specific parameters:**

**`format.number`:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `prefix` | string | `""` | Text before value |
| `suffix` | string | `""` | Text after value |
| `thousands_sep` | string | `""` | Thousands separator |
| `decimal_sep` | string | `"."` | Decimal separator |
| `precision` | int | `0` | Decimal places |
| `multiplier` | number | `1` | Multiply before display |

**`format.date`:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `format` | string | `"%Y-%m-%d"` | strftime pattern |

**`format.boolean`:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `true_value` | string | `"true"` | Display for true |
| `false_value` | string | `"false"` | Display for false |
| `null_value` | string | `""` | Display for null |

**`format.text`:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `transform` | string | - | `"uppercase"`, `"lowercase"`, `"capitalize"` |
| `max_length` | int | - | Truncate if exceeded |
| `ellipsis` | string | `"..."` | Suffix when truncated |

**`format.enum`:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `source` | object | - | Mapping or datagrid reference |
| `default` | string | `""` | Label for unknown values |

### Conditions

Conditions determine when a rule applies.

**Syntax:**

```python
if <left> <operator> <right>
if <operand> <unary_operator>
```

**Operators:**

| Operator | Description | Example |
|----------|-------------|---------|
| `==` | Equal | `value == 0` |
| `!=` | Not equal | `value != ""` |
| `<` | Less than | `value < 0` |
| `<=` | Less or equal | `value <= 100` |
| `>` | Greater than | `value > 1000` |
| `>=` | Greater or equal | `value >= 0` |
| `contains` | String contains | `value contains "error"` |
| `startswith` | String starts with | `value startswith "ERR"` |
| `endswith` | String ends with | `value endswith ".pdf"` |
| `in` | Value in list | `value in ["A", "B", "C"]` |
| `between` | Value in range | `value between 0 and 100` |
| `isempty` | Is null/empty | `value isempty` |
| `isnotempty` | Is not null/empty | `value isnotempty` |

**Negation:**

Use `not` to negate any condition:

```python
style("error") if not value in ["valid", "approved"]
style("warning") if not value contains "OK"
```

**Case sensitivity:**

String comparisons are case-insensitive by default. Use `(case)` modifier for case-sensitive:

```python
style("error") if value == "Error" (case)
style("warning") if value contains "WARN" (case)
```

### References

References allow comparing with values from other cells.

**Syntax:**

| Reference | Description | Example |
|-----------|-------------|---------|
| `value` | Current cell value | `value < 0` |
| `col.<name>` | Value from another column (same row) | `value > col.budget` |
| `col."<name>"` | Column with spaces | `value > col."max amount"` |
| `row.<index>` | Value from another row (same column) | `value != row.0` |
| `cell.<col>-<row>` | Specific cell by coordinates | `value == cell.status-0` |

**Examples:**

```python
# Compare with another column
column amount:
    style("error") if value > col.budget
    style("warning") if value > col.budget * 0.9

# Compare with header row
column total:
    style("highlight") if value == row.0

# Compare with specific cell
column status:
    style("success") if value == cell.status-0
```

---

## Formal Grammar (EBNF)

```ebnf
// Top-level structure
program         : scope+

// Scopes
scope           : scope_header NEWLINE INDENT rule+ DEDENT
scope_header    : column_scope | row_scope | cell_scope
column_scope    : "column" column_name ":"
row_scope       : "row" INTEGER ":"
cell_scope      : "cell" cell_ref ":"
column_name     : NAME | QUOTED_STRING
cell_ref        : "(" column_name "," INTEGER ")" | CELL_ID

// Rules
rule            : (style_expr format_expr? | format_expr style_expr?) condition? NEWLINE
condition       : "if" comparison

// Comparisons
comparison      : "not"? (binary_comp | unary_comp) case_modifier?
binary_comp     : operand operator operand
                | operand "in" list
                | operand "between" operand "and" operand
unary_comp      : operand ("isempty" | "isnotempty")
case_modifier   : "(" "case" ")"

// Operators
operator        : "==" | "!=" | "<" | "<=" | ">" | ">="
                | "contains" | "startswith" | "endswith"

// Operands
operand         : value_ref | column_ref | row_ref | cell_ref_expr | literal | arithmetic
value_ref       : "value"
column_ref      : "col." (NAME | QUOTED_STRING)
row_ref         : "row." INTEGER
cell_ref_expr   : "cell." NAME "-" INTEGER
literal         : STRING | NUMBER | BOOLEAN
arithmetic      : operand ("*" | "/" | "+" | "-") operand
list            : "[" (literal ("," literal)*)? "]"

// Style expression
style_expr      : "style" "(" style_args ")"
style_args      : (QUOTED_STRING ("," style_kwargs)?) | style_kwargs
style_kwargs    : style_kwarg ("," style_kwarg)*
style_kwarg     : NAME "=" (QUOTED_STRING | BOOLEAN | NUMBER)

// Format expression
format_expr     : format_preset | format_typed
format_preset   : "format" "(" QUOTED_STRING ("," format_kwargs)? ")"
format_typed    : "format" "." FORMAT_TYPE "(" format_kwargs? ")"
format_kwargs   : format_kwarg ("," format_kwarg)*
format_kwarg    : NAME "=" (QUOTED_STRING | BOOLEAN | NUMBER | dict)
dict            : "{" (dict_entry ("," dict_entry)*)? "}"
dict_entry      : QUOTED_STRING ":" QUOTED_STRING

// Tokens
FORMAT_TYPE     : "number" | "date" | "boolean" | "text" | "enum"
NAME            : /[a-zA-Z_][a-zA-Z0-9_]*/
QUOTED_STRING   : /"[^"]*"/ | /'[^']*'/
INTEGER         : /[0-9]+/
NUMBER          : /[0-9]+(\.[0-9]+)?/
BOOLEAN         : "True" | "False" | "true" | "false"
CELL_ID         : /tcell_[a-zA-Z0-9_-]+/
NEWLINE         : /\n/
INDENT          : /^[ \t]+/
DEDENT          : // decrease in indentation
```

---

## Examples

### Basic Examples

**Highlight negative values:**

```python
column amount:
    style("error") if value < 0
```

**Format as currency:**

```python
column price:
    format("EUR")
```

**Conditional formatting with multiple rules:**

```python
column status:
    style("success") if value == "approved"
    style("warning") if value == "pending"
    style("error") if value == "rejected"
```

**Style header row:**

```python
row 0:
    style("neutral", bold=True)
```

### Advanced Examples

**Compare with another column:**

```python
column actual:
    style("error") if value > col.budget
    style("warning") if value > col.budget * 0.8
    style("success") if value <= col.budget * 0.8
```

**Multiple formatting on same column:**

```python
column amount:
    format("EUR")
    style("error") if value < 0
    style("success", bold=True) if value > 10000
```

**Complex conditions:**

```python
column score:
    style("error") if value between 0 and 30
    style("warning") if value between 31 and 70
    style("success") if value between 71 and 100

column category:
    style("primary") if value in ["A", "B", "C"]
    style("secondary") if not value in ["A", "B", "C"]

column name:
    style("info") if value startswith "VIP"
    style("neutral") if value isempty
```

**Enum formatting with display mapping:**

```python
column status:
    format.enum(source={"draft": "Brouillon", "pending": "En attente", "approved": "Approuvé"}, default="Inconnu")
```

**Complete example - Financial report:**

```python
# Header styling
row 0:
    style("neutral", bold=True)

# Amount column
column amount:
    format.number(precision=2, suffix=" €", thousands_sep=" ")
    style("error") if value < 0
    style("success") if value > col.target

# Percentage column
column progress:
    format("percentage")
    style("error") if value < 0.5
    style("warning") if value between 0.5 and 0.8
    style("success") if value > 0.8

# Status column
column status:
    format.enum(source={"draft": "Draft", "review": "In Review", "approved": "Approved", "rejected": "Rejected"})
    style("neutral") if value == "draft"
    style("info") if value == "review"
    style("success") if value == "approved"
    style("error") if value == "rejected"

# Date column
column created_at:
    format.date(format="%d %b %Y")

# Highlight specific cell
cell (amount, 10):
    style("accent", bold=True)
```

---

## Autocompletion

The DSL editor provides context-aware autocompletion to help users write rules efficiently.

### How It Works

```
┌─────────────────────────────────────────────────────────────┐
│                     CodeMirror Editor                        │
│                                                              │
│  User types: style("err|                                     │
│                        ▲                                     │
│                        │ cursor position                     │
└────────────────────────┼────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│              Autocompletion Request (HTMX)                   │
│  { "text": "style(\"err", "cursor": 11, "context": "..." }  │
└─────────────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│                   Python Backend                             │
│                                                              │
│  1. Parse partial input                                      │
│  2. Determine context (inside style(), first arg)           │
│  3. Filter matching presets: ["error"]                      │
│  4. Return suggestions                                       │
└─────────────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│              Suggestions Dropdown                            │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ error     - Red background for errors               │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘
```

### Completion Contexts

| Context | Trigger | Suggestions |
|---------|---------|-------------|
| **Scope keyword** | Start of line | `column`, `row`, `cell` |
| **Column name** | After `column ` | Column names from DataGrid |
| **Style preset** | Inside `style("` | Style presets |
| **Style parameter** | Inside `style(... , ` | `bold`, `italic`, `color`, etc. |
| **Format preset** | Inside `format("` | Format presets |
| **Format type** | After `format.` | `number`, `date`, `boolean`, `text`, `enum` |
| **Format parameter** | Inside `format.number(` | Type-specific params |
| **Operator** | After operand | `==`, `<`, `contains`, etc. |
| **Column reference** | After `col.` | Column names |
| **Keyword** | After condition | `if`, `not`, `and`, `or` |

### Example Completion Flow

```
User types     │ Suggestions
───────────────┼────────────────────────────────────
col            │ column
column         │ [column names from grid]
column amount: │ (new line, indent)
    st         │ style
    style(     │ "error", "warning", "success", ...
    style("e   │ "error"
    style("err │ "error" (filtered)
    style("error", │ bold=, italic=, color=, ...
    style("error", b │ bold=
    style("error", bold=│ True, False
    style("error", bold=True) │ format(, if
    style("error", bold=True) if │ value, col., row., not
    style("error", bold=True) if value │ ==, !=, <, >, in, ...
    style("error", bold=True) if value < │ [number input]
```

---

## CodeMirror Integration

### DaisyUI Theme

The editor uses DaisyUI CSS variables for consistent theming:

```javascript
import { EditorView } from '@codemirror/view'
import { HighlightStyle, syntaxHighlighting } from '@codemirror/language'
import { tags } from '@lezer/highlight'

// Editor theme (container, cursor, selection)
const daisyEditorTheme = EditorView.theme({
  '&': {
    backgroundColor: 'var(--color-base-100)',
    color: 'var(--color-base-content)',
    fontSize: '14px',
  },
  '.cm-content': {
    fontFamily: 'var(--font-mono, ui-monospace, monospace)',
    padding: '8px 0',
  },
  '.cm-line': {
    padding: '0 8px',
  },
  '.cm-cursor': {
    borderLeftColor: 'var(--color-primary)',
    borderLeftWidth: '2px',
  },
  '.cm-selectionBackground': {
    backgroundColor: 'color-mix(in srgb, var(--color-primary) 20%, transparent)',
  },
  '&.cm-focused .cm-selectionBackground': {
    backgroundColor: 'color-mix(in srgb, var(--color-primary) 30%, transparent)',
  },
  '.cm-gutters': {
    backgroundColor: 'var(--color-base-200)',
    color: 'var(--color-base-content)',
    opacity: '0.5',
    border: 'none',
  },
  '.cm-activeLineGutter': {
    backgroundColor: 'var(--color-base-300)',
  },
  '.cm-activeLine': {
    backgroundColor: 'color-mix(in srgb, var(--color-base-content) 5%, transparent)',
  },
})

// Syntax highlighting
const daisyHighlightStyle = HighlightStyle.define([
  // Keywords: column, row, cell, if, not, and, or
  { tag: tags.keyword, color: 'var(--color-primary)', fontWeight: 'bold' },

  // Functions: style, format
  { tag: tags.function(tags.variableName), color: 'var(--color-secondary)' },

  // Strings: "error", "EUR", "approved"
  { tag: tags.string, color: 'var(--color-success)' },

  // Numbers: 0, 100, 3.14
  { tag: tags.number, color: 'var(--color-accent)' },

  // Operators: ==, <, >, contains, in
  { tag: tags.operator, color: 'var(--color-warning)' },
  { tag: tags.compareOperator, color: 'var(--color-warning)' },

  // Booleans: True, False
  { tag: tags.bool, color: 'var(--color-info)' },

  // Property names: bold=, precision=
  { tag: tags.propertyName, color: 'var(--color-base-content)', opacity: '0.8' },

  // Comments (if supported later)
  { tag: tags.comment, color: 'var(--color-base-content)', opacity: '0.5', fontStyle: 'italic' },

  // Invalid/errors
  { tag: tags.invalid, color: 'var(--color-error)', textDecoration: 'underline wavy' },
])

// Combined extension
export const daisyTheme = [
  daisyEditorTheme,
  syntaxHighlighting(daisyHighlightStyle),
]
```

### Lezer Grammar (Translated from lark)

The lark grammar is translated to Lezer format for client-side parsing:

```javascript
// formatrules.grammar (Lezer)

@top Program { scope+ }

@skip { space | newline }

scope {
  scopeHeader ":" indent rule+ dedent
}

scopeHeader {
  ColumnScope | RowScope | CellScope
}

ColumnScope { kw<"column"> columnName }
RowScope { kw<"row"> Integer }
CellScope { kw<"cell"> cellRef }

columnName { Name | QuotedString }
cellRef { "(" columnName "," Integer ")" | CellId }

rule {
  (styleExpr formatExpr? | formatExpr styleExpr?) condition?
}

condition { kw<"if"> comparison }

comparison {
  not? (binaryComp | unaryComp) caseModifier?
}

not { kw<"not"> }

binaryComp {
  operand compareOp operand |
  operand kw<"in"> list |
  operand kw<"between"> operand kw<"and"> operand
}

unaryComp {
  operand (kw<"isempty"> | kw<"isnotempty">)
}

caseModifier { "(" kw<"case"> ")" }

compareOp { "==" | "!=" | "<" | "<=" | ">" | ">=" | kw<"contains"> | kw<"startswith"> | kw<"endswith"> }

operand {
  ValueRef | ColumnRef | RowRef | CellRefExpr | literal | ArithmeticExpr
}

ValueRef { kw<"value"> }
ColumnRef { kw<"col"> "." (Name | QuotedString) }
RowRef { kw<"row"> "." Integer }
CellRefExpr { kw<"cell"> "." Name "-" Integer }

literal { QuotedString | Number | Boolean }

ArithmeticExpr { operand !arith arithmeticOp operand }
arithmeticOp { "*" | "/" | "+" | "-" }

list { "[" (literal ("," literal)*)? "]" }

styleExpr { kw<"style"> "(" styleArgs ")" }
styleArgs { (QuotedString ("," styleKwargs)?) | styleKwargs }
styleKwargs { styleKwarg ("," styleKwarg)* }
styleKwarg { Name "=" (QuotedString | Boolean | Number) }

formatExpr { formatPreset | formatTyped }
formatPreset { kw<"format"> "(" QuotedString ("," formatKwargs)? ")" }
formatTyped { kw<"format"> "." formatType "(" formatKwargs? ")" }
formatType { @specialize[@name=FormatType]<Name, "number" | "date" | "boolean" | "text" | "enum"> }
formatKwargs { formatKwarg ("," formatKwarg)* }
formatKwarg { Name "=" (QuotedString | Boolean | Number | Dict) }

Dict { "{" (dictEntry ("," dictEntry)*)? "}" }
dictEntry { QuotedString ":" QuotedString }

kw<term> { @specialize[@name={term}]<Name, term> }

@tokens {
  Name { @asciiLetter (@asciiLetter | @digit | "_")* }
  QuotedString { '"' (!["\\] | "\\" _)* '"' | "'" (!['\\] | "\\" _)* "'" }
  Integer { @digit+ }
  Number { @digit+ ("." @digit+)? }
  Boolean { "True" | "False" | "true" | "false" }
  CellId { "tcell_" ((@asciiLetter | @digit | "_" | "-"))+ }

  space { " " | "\t" }
  newline { "\n" | "\r\n" }

  @precedence { CellId, Name }
  @precedence { Number, Integer }
}

@precedence {
  arith @left
}

@detectDelim
```

### Editor Setup

```javascript
import { EditorState } from '@codemirror/state'
import { EditorView, keymap, lineNumbers, drawSelection } from '@codemirror/view'
import { defaultKeymap, indentWithTab } from '@codemirror/commands'
import { indentOnInput, bracketMatching } from '@codemirror/language'
import { autocompletion } from '@codemirror/autocomplete'
import { linter } from '@codemirror/lint'

import { formatRulesLanguage } from './formatrules-lang'  // Generated from grammar
import { daisyTheme } from './daisy-theme'
import { formatRulesCompletion } from './completion'
import { formatRulesLinter } from './linter'

function createFormatRulesEditor(container, initialValue, options = {}) {
  const { onChange, columns = [], stylePresets = [], formatPresets = [] } = options

  const state = EditorState.create({
    doc: initialValue,
    extensions: [
      lineNumbers(),
      drawSelection(),
      indentOnInput(),
      bracketMatching(),
      keymap.of([...defaultKeymap, indentWithTab]),

      // Language support
      formatRulesLanguage(),

      // Theme
      daisyTheme,

      // Autocompletion with context
      autocompletion({
        override: [formatRulesCompletion({ columns, stylePresets, formatPresets })],
      }),

      // Linting (validation)
      linter(formatRulesLinter()),

      // Change callback
      EditorView.updateListener.of((update) => {
        if (update.docChanged && onChange) {
          onChange(update.state.doc.toString())
        }
      }),
    ],
  })

  return new EditorView({
    state,
    parent: container,
  })
}
```

---

## Technical Choices

### Why lark (not pyparsing)

Both `lark` and `pyparsing` are mature Python parsing libraries. We chose **lark** for the following reasons:

| Criterion | lark | pyparsing |
|-----------|------|-----------|
| **Grammar definition** | Declarative EBNF string | Python combinators |
| **Portability to Lezer** | Direct translation possible | Manual rewrite required |
| **Grammar readability** | Standard BNF-like notation | Embedded in Python code |
| **Maintenance** | Single grammar source | Two separate grammars to sync |

**The key factor**: We need the same grammar for both:
1. **Server-side** (Python): Validation and execution
2. **Client-side** (JavaScript): Syntax highlighting and autocompletion

With lark's declarative EBNF grammar, we can translate it to Lezer (CodeMirror 6's parser system) with minimal effort. With pyparsing, the grammar is embedded in Python logic, making extraction and translation significantly harder.

**Example comparison:**

```python
# lark - declarative grammar (easy to translate)
grammar = """
    scope: "column" NAME ":" NEWLINE INDENT rule+ DEDENT
    NAME: /[a-zA-Z_][a-zA-Z0-9_]*/
"""
```

```javascript
// Lezer - similar structure
@top Program { scope+ }
scope { "column" Name ":" newline indent rule+ dedent }
Name { @asciiLetter (@asciiLetter | @digit | "_")* }
```

```python
# pyparsing - grammar in Python code (hard to translate)
NAME = Word(alphas, alphanums + "_")
scope = Keyword("column") + NAME + ":" + LineEnd() + IndentedBlock(rule)
```

### Why CodeMirror 6

For the web-based DSL editor, we chose **CodeMirror 6**:

| Criterion | CodeMirror 6 | Monaco | Ace |
|-----------|--------------|--------|-----|
| **Bundle size** | ~150KB | ~2MB | ~300KB |
| **Custom languages** | Lezer parser | Monarch tokenizer | Custom modes |
| **Theming** | CSS variables | JSON themes | CSS |
| **DaisyUI integration** | Native (CSS vars) | Requires mapping | Partial |
| **Architecture** | Modern, modular | Monolithic | Legacy |

CodeMirror 6's use of CSS variables makes it trivial to integrate with DaisyUI's theming system, ensuring visual consistency across the application.

### Shared Grammar Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Grammar Source (EBNF)                     │
│                    lark format in Python                     │
└─────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┴───────────────┐
              ▼                               ▼
┌─────────────────────────┐     ┌─────────────────────────────┐
│     Python (Server)      │     │    JavaScript (Client)       │
│     lark parser          │     │    Lezer parser              │
│                          │     │    (translated from lark)    │
│  • Validation            │     │                              │
│  • Execution             │     │  • Syntax highlighting       │
│  • Error messages        │     │  • Autocompletion            │
│  • Preset resolution     │     │  • Error markers             │
└─────────────────────────┘     └─────────────────────────────┘
```

---

## Autocompletion API

### Overview

The autocompletion system provides context-aware suggestions for the DSL editor. Intelligence runs **server-side** via a REST API, while the editor (CodeMirror 5) handles display and user interaction.

### Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    CodeMirror Editor                        │
│  User types: style("err|                                    │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│              REST API: /myfasthtml/autocompletion           │
│  Request: { "text": "...", "cursor": {"line": 1, "ch": 15} }│
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│              Python: Autocompletion Engine                  │
│                                                             │
│  1. Detect SCOPE from previous lines                        │
│     - Find scope declaration (column/row/cell)              │
│     - Extract scope name (e.g., "amount" for column)        │
│                                                             │
│  2. Detect COMPLETION CONTEXT at cursor position            │
│     - Analyze current line up to cursor                     │
│     - Determine what kind of token is expected              │
│                                                             │
│  3. Generate suggestions                                    │
│     - Query DatagridMetadataProvider if needed              │
│     - Filter suggestions by prefix                          │
│                                                             │
│  4. Return CodeMirror-compatible response                   │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│  Response: {                                                │
│    "from": {"line": 1, "ch": 12},                           │
│    "to": {"line": 1, "ch": 15},                             │
│    "suggestions": [{"label": "error", "detail": "..."}]     │
│  }                                                          │
└─────────────────────────────────────────────────────────────┘
```

### Scope Detection

Before determining completion context, the engine must identify the **current scope** by scanning backwards from the cursor position to find the most recent scope declaration.

**Scope Types:**

| Scope Type | Pattern | Extracted Info |
|------------|---------|----------------|
| Column | `column <name>:` or `column "<name>":` | Column name |
| Row | `row <index>:` | Row index |
| Cell | `cell (<col>, <row>):` | Column name + row index |

**Algorithm:**

1. Start from cursor line
2. Scan backwards for a non-indented line matching scope pattern
3. Extract scope type and parameters
4. If no scope found, cursor is at top level (suggest scope keywords)

**Example:**

```python
column status:           # ← Scope: ColumnScope("status")
    style("error") if |  # ← Cursor here
```

When the engine needs column values for `OPERATOR_VALUE` context, it uses the detected scope to call `provider.get_column_values("status")`.

### DatagridMetadataProvider

A helper class providing access to DataGrid metadata for context-aware suggestions:

```python
class DatagridMetadataProvider:
    """Provides DataGrid metadata for autocompletion."""

    def get_tables(self) -> list[str]:
        """List of available DataGrids (namespace.name format)."""
        ...

    def get_columns(self, table_name: str) -> list[str]:
        """Column names for a specific DataGrid."""
        ...

    def get_column_values(self, column_name: str) -> list[Any]:
        """Distinct values for a column in the current scope."""
        ...

    def get_row_count(self, table_name: str) -> int:
        """Number of rows in a DataGrid."""
        ...
```

**Notes:**
- DataGrid names follow the pattern `namespace.name` (multi-level namespaces supported)
- The provider is passed at initialization, not with each API call
- Column values are fetched lazily when the scope is detected

### API Interface

**Request:**

```json
{
  "text": "column amount:\n    style(\"err",
  "cursor": {"line": 1, "ch": 15}
}
```

- `text`: Full DSL document content
- `cursor.line`: 0-based line number
- `cursor.ch`: 0-based character position in line

**Response:**

```json
{
  "from": {"line": 1, "ch": 12},
  "to": {"line": 1, "ch": 15},
  "suggestions": [
    {"label": "error", "detail": "Red background for errors"},
    {"label": "warning", "detail": "Yellow background for warnings"}
  ]
}
```

- `from`/`to`: Range to replace (word boundaries using delimiters)
- `suggestions`: List of completion items with label and optional detail

### Completion Contexts

| Context | Trigger | Suggestions |
|---------|---------|-------------|
| `SCOPE_KEYWORD` | Start of non-indented line | `column`, `row`, `cell` |
| `COLUMN_NAME` | After `column ` | Column names from DataGrid |
| `ROW_INDEX` | After `row ` | First 10 indices + last index |
| `CELL_START` | After `cell ` | `(` |
| `CELL_COLUMN` | After `cell (` | Column names |
| `CELL_ROW` | After `cell (col, ` | First 10 indices + last index |
| `RULE_START` | Start of indented line (after scope) | `style(`, `format(`, `format.` |
| `STYLE_ARGS` | After `style(` (no quote) | Presets with quotes + named params (`bold=`, `color=`, ...) |
| `STYLE_PRESET` | Inside `style("` | Style presets |
| `STYLE_PARAM` | After comma in `style(...)` | `bold=`, `italic=`, `underline=`, `strikethrough=`, `color=`, `background_color=`, `font_size=` |
| `FORMAT_PRESET` | Inside `format("` | Format presets |
| `FORMAT_TYPE` | After `format.` | `number`, `date`, `boolean`, `text`, `enum` |
| `FORMAT_PARAM_DATE` | Inside `format.date(` | `format=` |
| `FORMAT_PARAM_TEXT` | Inside `format.text(` | `transform=`, `max_length=`, `ellipsis=` |
| `AFTER_STYLE_OR_FORMAT` | After `)` of style/format | `style(`, `format(`, `format.`, `if` |
| `CONDITION_START` | After `if ` | `value`, `col.`, `not` |
| `CONDITION_AFTER_NOT` | After `if not ` | `value`, `col.` |
| `COLUMN_REF` | After `col.` | Column names |
| `COLUMN_REF_QUOTED` | After `col."` | Column names (with closing quote) |
| `OPERATOR` | After operand (`value`, literal, ref) | `==`, `!=`, `<`, `<=`, `>`, `>=`, `contains`, `startswith`, `endswith`, `in`, `between`, `isempty`, `isnotempty` |
| `OPERATOR_VALUE` | After comparison operator | `col.`, `True`, `False`, column values (from detected scope) |
| `BETWEEN_AND` | After `between X ` | `and` |
| `BETWEEN_VALUE` | After `between X and ` | Same as `OPERATOR_VALUE` |
| `IN_LIST_START` | After `in ` | `[` |
| `IN_LIST_VALUE` | Inside `[` or after `,` in list | Column values (from detected scope) |
| `BOOLEAN_VALUE` | After `bold=`, `italic=`, etc. | `True`, `False` |
| `COLOR_VALUE` | After `color=`, `background_color=` | CSS colors + DaisyUI variables |
| `DATE_FORMAT_VALUE` | After `format=` in format.date | `"%Y-%m-%d"`, `"%d/%m/%Y"`, `"%m/%d/%Y"`, `"%d %b %Y"` |
| `TRANSFORM_VALUE` | After `transform=` | `"uppercase"`, `"lowercase"`, `"capitalize"` |
| `COMMENT` | After `#` | No suggestions |

### Context Detection Flow

```
column |           → SCOPE_KEYWORD: column, row, cell
column a|          → COLUMN_NAME: amount, active, ...
column amount:
    |              → RULE_START: style(, format(, format.
    style(|        → STYLE_ARGS: "error", "warning", ..., bold=, color=, ...
    style("|       → STYLE_PRESET: error, warning, success, ...
    style("e|      → STYLE_PRESET (filtered): error
    style("error", |  → STYLE_PARAM: bold=, italic=, color=, ...
    style("error", bold=|  → BOOLEAN_VALUE: True, False
    style("error", color=|  → COLOR_VALUE: red, blue, var(--color-primary), ...
    style("error")|  → AFTER_STYLE_OR_FORMAT: format(, format., if
    style("error") if |  → CONDITION_START: value, col., not
    style("error") if not |  → CONDITION_AFTER_NOT: value, col.
    style("error") if value |  → OPERATOR: ==, <, >, contains, in, ...
    style("error") if value == |  → OPERATOR_VALUE: col., True, False, "draft", "pending", ...
    style("error") if value in |  → IN_LIST_START: [
    style("error") if value in [|  → IN_LIST_VALUE: "draft", "pending", ...
    style("error") if value in ["draft", |  → IN_LIST_VALUE: "pending", ...
    style("error") if value between |  → OPERATOR_VALUE
    style("error") if value between 0 |  → BETWEEN_AND: and
    style("error") if value between 0 and |  → BETWEEN_VALUE

row |              → ROW_INDEX: 0, 1, 2, ..., 9, 149 (if 150 rows)
row 0:
    |              → RULE_START

cell |             → CELL_START: (
cell (|            → CELL_COLUMN: amount, status, ...
cell (amount, |    → CELL_ROW: 0, 1, 2, ..., 9, 149
cell (amount, 3):
    |              → RULE_START

    format.|       → FORMAT_TYPE: number, date, boolean, text, enum
    format.date(|  → FORMAT_PARAM_DATE: format=
    format.date(format=|  → DATE_FORMAT_VALUE: "%Y-%m-%d", "%d/%m/%Y", ...
    format.text(|  → FORMAT_PARAM_TEXT: transform=, max_length=, ellipsis=
    format.text(transform=|  → TRANSFORM_VALUE: "uppercase", "lowercase", "capitalize"
```

### Suggestions Data

**Style Presets (DaisyUI 5):**

| Label | Detail |
|-------|--------|
| `primary` | Primary theme color |
| `secondary` | Secondary theme color |
| `accent` | Accent theme color |
| `neutral` | Neutral theme color |
| `info` | Info (blue) |
| `success` | Success (green) |
| `warning` | Warning (yellow) |
| `error` | Error (red) |

**Format Presets:**

| Label | Detail |
|-------|--------|
| `EUR` | Euro currency (1 234,56 €) |
| `USD` | US Dollar ($1,234.56) |
| `percentage` | Percentage (×100, adds %) |
| `short_date` | DD/MM/YYYY |
| `iso_date` | YYYY-MM-DD |
| `yes_no` | Yes/No |

**CSS Colors (subset):**

`red`, `blue`, `green`, `yellow`, `orange`, `purple`, `pink`, `gray`, `black`, `white`

**DaisyUI Color Variables:**

`var(--color-primary)`, `var(--color-secondary)`, `var(--color-accent)`, `var(--color-neutral)`, `var(--color-info)`, `var(--color-success)`, `var(--color-warning)`, `var(--color-error)`, `var(--color-base-100)`, `var(--color-base-200)`, `var(--color-base-300)`, `var(--color-base-content)`

**Date Format Patterns:**

| Label | Detail |
|-------|--------|
| `"%Y-%m-%d"` | ISO format (2026-01-29) |
| `"%d/%m/%Y"` | European (29/01/2026) |
| `"%m/%d/%Y"` | US format (01/29/2026) |
| `"%d %b %Y"` | Short month (29 Jan 2026) |
| `"%d %B %Y"` | Full month (29 January 2026) |

### Word Boundary Detection

To determine the range to replace (`from`/`to`), use delimiters:
- Quotes: `"`, `'`
- Parentheses: `(`, `)`
- Brackets: `[`, `]`
- Braces: `{`, `}`
- Operators: `=`, `,`, `:`, `<`, `>`, `!`
- Whitespace: space, tab, newline

### Exclusions (Not Implemented)

The following features are excluded from autocompletion for simplicity:

- `(case)` modifier for case-sensitive comparisons
- Arithmetic expressions (`col.budget * 0.9`)
- `format.enum` source configuration
- `format.number` parameters (free input)
- `format.boolean` parameters (free input)

---

## Implementation Status

| Component | Status | Location |
|-----------|--------|----------|
| DSL Grammar (lark) | :white_check_mark: Implemented | `src/myfasthtml/core/formatting/dsl/grammar.py` |
| DSL Parser | :white_check_mark: Implemented | `src/myfasthtml/core/formatting/dsl/parser.py` |
| DSL Transformer | :white_check_mark: Implemented | `src/myfasthtml/core/formatting/dsl/transformer.py` |
| Scope Dataclasses | :white_check_mark: Implemented | `src/myfasthtml/core/formatting/dsl/scopes.py` |
| Exceptions | :white_check_mark: Implemented | `src/myfasthtml/core/formatting/dsl/exceptions.py` |
| Public API (`parse_dsl()`) | :white_check_mark: Implemented | `src/myfasthtml/core/formatting/dsl/__init__.py` |
| Unit Tests (Parser) | :white_check_mark: ~35 tests | `tests/core/formatting/test_dsl_parser.py` |
| **Autocompletion** | | |
| DatagridMetadataProvider | :x: Not implemented | `src/myfasthtml/core/formatting/dsl/completion.py` |
| Scope Detector | :x: Not implemented | `src/myfasthtml/core/formatting/dsl/completion.py` |
| Context Detector | :x: Not implemented | `src/myfasthtml/core/formatting/dsl/completion.py` |
| Suggestions Generator | :x: Not implemented | `src/myfasthtml/core/formatting/dsl/completion.py` |
| REST Endpoint | :x: Not implemented | `/myfasthtml/autocompletion` |
| Unit Tests (Completion) | :x: Not implemented | `tests/core/formatting/test_dsl_completion.py` |
| **Client-side** | | |
| Lezer Grammar | :x: Not implemented | `static/js/formatrules.grammar` |
| CodeMirror Extension | :x: Not implemented | `static/js/formatrules-editor.js` |
| DaisyUI Theme | :x: Not implemented | `static/js/daisy-theme.js` |

---

## Implementation Notes

### Dependencies

The DSL module requires:
- `lark` - Python parsing library (added to `pyproject.toml`)

### Key Implementation Details

**Indentation handling (lark Indenter):**

The DSL uses Python-style indentation. Lark's `Indenter` class requires the newline token (`_NL`) to include trailing whitespace so it can detect indentation levels:

```
_NL: /(\r?\n[\t ]*)+/
```

This means `_NL` captures the newline AND the following spaces/tabs. The Indenter uses `token.rsplit('\n', 1)[1]` to extract the indentation string.

**Comment handling:**

Comments (`# ...`) are pre-processed in `parser.py` before parsing. Comment lines are replaced with empty strings to preserve line numbers in error messages:

```python
lines = ["" if line.strip().startswith("#") else line for line in lines]
```

The grammar's `%ignore COMMENT` directive handles inline comments (at end of lines).

**Module Structure:**

```
src/myfasthtml/core/formatting/dsl/
├── __init__.py          # Public API: parse_dsl()
├── grammar.py           # Lark EBNF grammar string
├── parser.py            # DSLParser class with Indenter
├── transformer.py       # AST → dataclass conversion
├── scopes.py            # ColumnScope, RowScope, CellScope, ScopedRule
└── exceptions.py        # DSLSyntaxError, DSLValidationError
```

**Output structure:**

`parse_dsl()` returns a list of `ScopedRule` objects:

```python
@dataclass
class ScopedRule:
    scope: ColumnScope | RowScope | CellScope
    rule: FormatRule  # From core/formatting/dataclasses.py
```

### Next Steps

1. ~~**Add `lark` to dependencies** in `pyproject.toml`~~ Done
2. **Implement autocompletion API**:
   - `DatagridMetadataProvider` class
   - Scope detection (column/row/cell)
   - Context detection
   - Suggestions generation
   - REST endpoint `/myfasthtml/autocompletion`
3. **Translate lark grammar to Lezer** for client-side parsing
4. **Build CodeMirror extension** with DaisyUI theme
5. ~~**Integrate with DataGrid** - connect DSL output to formatting engine~~ Done