Added syntax colorization
This commit is contained in:
@@ -700,11 +700,99 @@ export const daisyTheme = [
|
||||
]
|
||||
```
|
||||
|
||||
### Lezer Grammar (Translated from lark)
|
||||
### CodeMirror Simple Mode (Generated from lark)
|
||||
|
||||
The lark grammar is translated to Lezer format for client-side parsing:
|
||||
The lark grammar terminals are extracted and converted to CodeMirror 5 Simple Mode format for syntax highlighting:
|
||||
|
||||
```javascript
|
||||
// Generated from lark grammar terminals
|
||||
CodeMirror.defineSimpleMode("formatting-dsl", {
|
||||
start: [
|
||||
// Comments
|
||||
{regex: /#.*/, token: "comment"},
|
||||
|
||||
// Keywords
|
||||
{regex: /\b(?:column|row|cell|if|not|and|or|in|between|case)\b/, token: "keyword"},
|
||||
|
||||
// Built-in functions
|
||||
{regex: /\b(?:style|format)\b/, token: "builtin"},
|
||||
|
||||
// Operators
|
||||
{regex: /\b(?:contains|startswith|endswith|isempty|isnotempty)\b/, token: "operator"},
|
||||
{regex: /==|!=|<=|>=|<|>/, token: "operator"},
|
||||
|
||||
// References
|
||||
{regex: /\b(?:value|col)\b/, token: "variable-2"},
|
||||
|
||||
// Booleans
|
||||
{regex: /\b(?:True|False|true|false)\b/, token: "atom"},
|
||||
|
||||
// Numbers
|
||||
{regex: /\b\d+(?:\.\d+)?\b/, token: "number"},
|
||||
|
||||
// Strings
|
||||
{regex: /"(?:[^\\]|\\.)*?"/, token: "string"},
|
||||
{regex: /'(?:[^\\]|\\.)*?'/, token: "string"},
|
||||
|
||||
// Cell IDs
|
||||
{regex: /\btcell_[a-zA-Z0-9_-]+\b/, token: "variable-3"},
|
||||
]
|
||||
});
|
||||
```
|
||||
|
||||
**Token classes**:
|
||||
- `comment` - Comments (`#`)
|
||||
- `keyword` - Keywords (`column`, `row`, `cell`, `if`, `not`, `and`, `or`, `in`, `between`, `case`)
|
||||
- `builtin` - Built-in functions (`style`, `format`)
|
||||
- `operator` - Comparison/string operators (`==`, `<`, `contains`, etc.)
|
||||
- `variable-2` - Special variables (`value`, `col`)
|
||||
- `atom` - Literals (`True`, `False`)
|
||||
- `number` - Numeric literals
|
||||
- `string` - String literals
|
||||
- `variable-3` - Cell IDs (`tcell_*`)
|
||||
|
||||
These classes are styled via CSS using DaisyUI color variables for automatic theme support.
|
||||
|
||||
### Editor Setup (CodeMirror 5)
|
||||
|
||||
```javascript
|
||||
function initDslEditor(config) {
|
||||
const editor = CodeMirror(container, {
|
||||
value: initialValue,
|
||||
mode: "formatting-dsl", // Use generated Simple Mode
|
||||
lineNumbers: true,
|
||||
extraKeys: {
|
||||
"Ctrl-Space": "autocomplete"
|
||||
},
|
||||
hintOptions: {
|
||||
hint: dslHint, // Server-side completions
|
||||
completeSingle: false
|
||||
},
|
||||
gutters: ["CodeMirror-linenumbers", "CodeMirror-lint-markers"],
|
||||
lint: {
|
||||
getAnnotations: dslLint, // Server-side validation
|
||||
async: true
|
||||
}
|
||||
});
|
||||
|
||||
return editor;
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: The Simple Mode provides instant syntax highlighting (approximative, lexer-level). Server-side validation via `/myfasthtml/validations` provides accurate error reporting.
|
||||
|
||||
---
|
||||
|
||||
## CodeMirror Integration (Deprecated - CodeMirror 6)
|
||||
|
||||
The following sections describe the original approach using CodeMirror 6 + Lezer, which was abandoned due to bundler requirements incompatible with FastHTML.
|
||||
|
||||
### ~~Lezer Grammar (Translated from lark)~~ (Deprecated)
|
||||
|
||||
~~The lark grammar is translated to Lezer format for client-side parsing:~~
|
||||
|
||||
```javascript
|
||||
// DEPRECATED: CodeMirror 6 + Lezer approach (requires bundler)
|
||||
// formatrules.grammar (Lezer)
|
||||
|
||||
@top Program { scope+ }
|
||||
@@ -807,9 +895,10 @@ kw<term> { @specialize[@name={term}]<Name, term> }
|
||||
@detectDelim
|
||||
```
|
||||
|
||||
### Editor Setup
|
||||
### ~~Editor Setup~~ (Deprecated - CodeMirror 6)
|
||||
|
||||
```javascript
|
||||
// DEPRECATED: CodeMirror 6 approach (requires bundler, incompatible with FastHTML)
|
||||
import { EditorState } from '@codemirror/state'
|
||||
import { EditorView, keymap, lineNumbers, drawSelection } from '@codemirror/view'
|
||||
import { defaultKeymap, indentWithTab } from '@codemirror/commands'
|
||||
@@ -875,54 +964,31 @@ Both `lark` and `pyparsing` are mature Python parsing libraries. We chose **lark
|
||||
| Criterion | lark | pyparsing |
|
||||
|-----------|------|-----------|
|
||||
| **Grammar definition** | Declarative EBNF string | Python combinators |
|
||||
| **Portability to Lezer** | Direct translation possible | Manual rewrite required |
|
||||
| **Regex extraction** | Easy to extract terminals | Embedded in Python logic |
|
||||
| **Grammar readability** | Standard BNF-like notation | Embedded in Python code |
|
||||
| **Maintenance** | Single grammar source | Two separate grammars to sync |
|
||||
|
||||
**The key factor**: We need the same grammar for both:
|
||||
1. **Server-side** (Python): Validation and execution
|
||||
2. **Client-side** (JavaScript): Syntax highlighting and autocompletion
|
||||
2. **Client-side** (JavaScript): Syntax highlighting
|
||||
|
||||
With lark's declarative EBNF grammar, we can translate it to Lezer (CodeMirror 6's parser system) with minimal effort. With pyparsing, the grammar is embedded in Python logic, making extraction and translation significantly harder.
|
||||
With lark's declarative EBNF grammar, we can extract terminal regex patterns and translate them to CodeMirror Simple Mode for client-side syntax highlighting. With pyparsing, the grammar is embedded in Python logic, making extraction significantly harder.
|
||||
|
||||
**Example comparison:**
|
||||
### Why CodeMirror 5 (not 6)
|
||||
|
||||
```python
|
||||
# lark - declarative grammar (easy to translate)
|
||||
grammar = """
|
||||
scope: "column" NAME ":" NEWLINE INDENT rule+ DEDENT
|
||||
NAME: /[a-zA-Z_][a-zA-Z0-9_]*/
|
||||
"""
|
||||
```
|
||||
For the web-based DSL editor, we chose **CodeMirror 5**:
|
||||
|
||||
```javascript
|
||||
// Lezer - similar structure
|
||||
@top Program { scope+ }
|
||||
scope { "column" Name ":" newline indent rule+ dedent }
|
||||
Name { @asciiLetter (@asciiLetter | @digit | "_")* }
|
||||
```
|
||||
| Criterion | CodeMirror 5 | CodeMirror 6 | Monaco | Ace |
|
||||
|-----------|--------------|--------------|--------|-----|
|
||||
| **Bundle requirements** | CDN ready | Requires bundler | ~2MB | ~300KB |
|
||||
| **FastHTML compatibility** | Direct `<script>` include | Not compatible | Requires build | Partial |
|
||||
| **Custom languages** | Simple Mode (regex) | Lezer parser | Monarch tokenizer | Custom modes |
|
||||
| **Theming** | CSS classes | CSS variables | JSON themes | CSS |
|
||||
| **DaisyUI integration** | CSS classes + variables | Native (CSS vars) | Requires mapping | Partial |
|
||||
|
||||
```python
|
||||
# pyparsing - grammar in Python code (hard to translate)
|
||||
NAME = Word(alphas, alphanums + "_")
|
||||
scope = Keyword("column") + NAME + ":" + LineEnd() + IndentedBlock(rule)
|
||||
```
|
||||
**Key constraint**: CodeMirror 6 requires a bundler (Webpack, Rollup, etc.) which is incompatible with FastHTML's direct script inclusion approach. CodeMirror 5's Simple Mode addon allows us to define syntax highlighting using regex patterns extracted from the lark grammar.
|
||||
|
||||
### Why CodeMirror 6
|
||||
|
||||
For the web-based DSL editor, we chose **CodeMirror 6**:
|
||||
|
||||
| Criterion | CodeMirror 6 | Monaco | Ace |
|
||||
|-----------|--------------|--------|-----|
|
||||
| **Bundle size** | ~150KB | ~2MB | ~300KB |
|
||||
| **Custom languages** | Lezer parser | Monarch tokenizer | Custom modes |
|
||||
| **Theming** | CSS variables | JSON themes | CSS |
|
||||
| **DaisyUI integration** | Native (CSS vars) | Requires mapping | Partial |
|
||||
| **Architecture** | Modern, modular | Monolithic | Legacy |
|
||||
|
||||
CodeMirror 6's use of CSS variables makes it trivial to integrate with DaisyUI's theming system, ensuring visual consistency across the application.
|
||||
|
||||
### Shared Grammar Architecture
|
||||
### Grammar Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
@@ -934,15 +1000,18 @@ CodeMirror 6's use of CSS variables makes it trivial to integrate with DaisyUI's
|
||||
▼ ▼
|
||||
┌─────────────────────────┐ ┌─────────────────────────────┐
|
||||
│ Python (Server) │ │ JavaScript (Client) │
|
||||
│ lark parser │ │ Lezer parser │
|
||||
│ │ │ (translated from lark) │
|
||||
│ lark parser │ │ CodeMirror Simple Mode │
|
||||
│ │ │ (regex from terminals) │
|
||||
│ • Validation │ │ │
|
||||
│ • Execution │ │ • Syntax highlighting │
|
||||
│ • Error messages │ │ • Autocompletion │
|
||||
│ • Preset resolution │ │ • Error markers │
|
||||
│ • Error messages │ │ (approximative, instant) │
|
||||
│ • Preset resolution │ │ │
|
||||
│ • Autocompletion logic │ │ • Autocompletion display │
|
||||
└─────────────────────────┘ └─────────────────────────────┘
|
||||
```
|
||||
|
||||
**Note**: Client-side syntax highlighting is **approximative** (lexer-level only) but **instantaneous**. Server-side validation provides accurate error reporting with debouncing.
|
||||
|
||||
---
|
||||
|
||||
## Autocompletion API
|
||||
@@ -1258,12 +1327,13 @@ The following features are excluded from autocompletion for simplicity:
|
||||
| Unit Tests (Completion) | :white_check_mark: ~50 tests | `tests/core/formatting/dsl/test_completion.py` |
|
||||
| REST Endpoint | :white_check_mark: Implemented | `src/myfasthtml/core/utils.py` → `/myfasthtml/completions` |
|
||||
| **Client-side** | | |
|
||||
| Lark to Lezer converter | :white_check_mark: Implemented | `src/myfasthtml/core/dsl/lark_to_lezer.py` |
|
||||
| Lark to Simple Mode converter | :o: To implement | - |
|
||||
| CodeMirror 5 assets | :white_check_mark: Implemented | `assets/codemirror.min.js`, `show-hint.min.js` |
|
||||
| DslEditor control | :white_check_mark: Implemented | `src/myfasthtml/controls/DslEditor.py` |
|
||||
| initDslEditor() JS | :white_check_mark: Implemented | `assets/myfasthtml.js` (static completions only) |
|
||||
| Dynamic completions (server calls) | :white_check_mark: Implemented | `assets/myfasthtml.js` → `/myfasthtml/completions` |
|
||||
| DaisyUI Theme | :o: Deferred | - |
|
||||
| DaisyUI Theme (CSS styling) | :o: To implement | `myfasthtml.css` |
|
||||
| Syntax highlighting (Simple Mode) | :o: To implement | `myfasthtml.js` |
|
||||
| **Syntax Validation (Linting)** | | |
|
||||
| REST Endpoint | :white_check_mark: Implemented | `src/myfasthtml/core/utils.py` → `/myfasthtml/validations` |
|
||||
| CodeMirror lint integration | :white_check_mark: Implemented | `assets/myfasthtml.js` → `dslLint()` |
|
||||
@@ -1400,14 +1470,23 @@ These warnings would be non-blocking (DSL still parses) but displayed with a dif
|
||||
|
||||
1. ~~**Add `lark` to dependencies** in `pyproject.toml`~~ Done
|
||||
2. ~~**Implement autocompletion API**~~ Done
|
||||
3. ~~**Translate lark grammar to Lezer**~~ Done (`lark_to_lezer.py`)
|
||||
4. ~~**Build CodeMirror extension**~~ Done (`DslEditor.py` + `initDslEditor()`)
|
||||
5. ~~**Integrate with DataGrid** - connect DSL output to formatting engine~~ Done
|
||||
3. ~~**Build CodeMirror 5 wrapper**~~ Done (`DslEditor.py` + `initDslEditor()`)
|
||||
4. ~~**Integrate with DataGrid** - connect DSL output to formatting engine~~ Done
|
||||
- `DataGridFormattingEditor.on_content_changed()` dispatches rules to column/row/cell formats
|
||||
- `DataGrid.mk_body_cell_content()` applies formatting via `FormattingEngine`
|
||||
6. ~~**Implement dynamic client-side completions**~~ Done
|
||||
5. ~~**Implement dynamic client-side completions**~~ Done
|
||||
- Engine ID: `DSLDefinition.get_id()` passed via `completionEngineId` in JS config
|
||||
- Trigger: Hybrid (Ctrl+Space + auto after `.` `(` `"` and space)
|
||||
- Files modified:
|
||||
- `src/myfasthtml/controls/DslEditor.py:134` - Added `completionEngineId`
|
||||
- `src/myfasthtml/assets/myfasthtml.js:2138-2300` - Async fetch to `/myfasthtml/completions`
|
||||
- `src/myfasthtml/controls/DslEditor.py:149` - Added `dslId`
|
||||
- `src/myfasthtml/assets/myfasthtml.js:2138-2349` - Async fetch to `/myfasthtml/completions`
|
||||
6. ~~**Implement validation (linting)**~~ Done
|
||||
- REST endpoint: `/myfasthtml/validations`
|
||||
- CodeMirror lint addon integration in `initDslEditor()`
|
||||
7. **DaisyUI Theme for CodeMirror** - To implement
|
||||
- Add CSS styles in `myfasthtml.css` mapping CodeMirror classes to DaisyUI variables
|
||||
- Ensures editor adapts to theme changes (dark mode, etc.)
|
||||
8. **Syntax highlighting (Simple Mode)** - To implement
|
||||
- Extract terminals from lark grammar → generate Simple Mode config
|
||||
- Add Simple Mode addon (~3KB) to assets
|
||||
- Register mode in `initDslEditor()`
|
||||
|
||||
Reference in New Issue
Block a user