Architecture¶

Technical overview of jsonatapy's architecture and design.

High-Level Overview¶

jsonatapy is a Rust-based Python extension implementing the JSONata query and transformation language.

Architecture Layers¶

┌─────────────────────────────────────────┐
│         Python Application              │
│         (User Code)                     │
└─────────────────┬───────────────────────┘
                  │
┌─────────────────▼───────────────────────┐
│    Python API (python/jsonatapy/)       │
│    - compile(), evaluate()              │
│    - JsonataExpression wrapper          │
│    - JsonataData wrapper                │
└─────────────────┬───────────────────────┘
                  │
┌─────────────────▼───────────────────────┐
│    PyO3 Bindings (src/lib.rs)           │
│    - Python↔Rust boundary               │
│    - Type conversions                   │
│    - Exception handling                 │
└─────────────────┬───────────────────────┘
                  │
┌─────────────────▼───────────────────────┐
│         Rust Core                       │
│    ┌─────────────────────────────┐     │
│    │  Parser (src/parser.rs)     │     │
│    │  - Tokenization             │     │
│    │  - Expression parsing       │     │
│    │  - AST generation           │     │
│    └────────────┬────────────────┘     │
│                 │                       │
│    ┌────────────▼────────────────┐     │
│    │  Evaluator (src/evaluator.rs)│    │
│    │  - Expression evaluation    │     │
│    │  - Context management       │     │
│    │  - Lambda execution         │     │
│    └────────────┬────────────────┘     │
│                 │                       │
│    ┌────────────▼────────────────┐     │
│    │  Functions (src/functions.rs)│    │
│    │  - Built-in functions       │     │
│    │  - Datetime (src/datetime.rs)│    │
│    │  - Signature validation     │     │
│    └─────────────────────────────┘     │
└─────────────────────────────────────────┘

Key Components¶

Python API Layer - User-facing Python interface
PyO3 Bindings - Python-Rust interop layer
Parser - Converts JSONata expressions to AST
Evaluator - Executes AST against data
Functions - Built-in function implementations

Module Structure¶

jsonatapy mirrors the structure of the JavaScript reference implementation for maintainability.

src/
├── lib.rs          # PyO3 bindings, Python API entry point
├── parser.rs       # Expression parser (mirrors parser.js)
├── ast.rs          # AST node definitions
├── evaluator.rs    # Main evaluation engine (mirrors jsonata.js)
├── functions.rs    # Built-in functions (mirrors functions.js)
├── datetime.rs     # Date/time functions (mirrors datetime.js)
├── signature.rs    # Function signature validation (mirrors signature.js)
└── value.rs        # JValue type system (custom to Rust)

Module Responsibilities¶

lib.rs - Python Bindings

#[pymodule]
fn _jsonatapy(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(compile, m)?)?;
    m.add_function(wrap_pyfunction!(evaluate, m)?)?;
    m.add_class::<JsonataExpression>()?;
    m.add_class::<JsonataData>()?;
    Ok(())
}

parser.rs - Lexical Analysis and Parsing - Tokenizes JSONata expression strings - Builds Abstract Syntax Tree (AST) - Handles operator precedence - Reports syntax errors

evaluator.rs - Expression Evaluation (~7500 lines) - Main evaluation engine - Context and scope management - Lambda storage and execution - Path traversal and predicate evaluation - Object construction

functions.rs - Built-in Functions - 40+ built-in JSONata functions - String manipulation ($uppercase, $lowercase, $substring, etc.) - Array operations ($map, $filter, $reduce, etc.) - Numeric functions ($sum, $average, $max, etc.) - Object functions ($keys, $values, $merge, etc.)

value.rs - Type System - JValue enum representing all JSONata types - Rc-wrapped for O(1) cloning - Conversion to/from Python types

JValue Type System¶

JValue Enum¶

jsonatapy uses a custom JValue type instead of serde_json::Value for performance:

pub enum JValue {
    Null,
    Bool(bool),
    Number(f64),
    String(Rc<str>),              // Rc-wrapped for O(1) clone
    Array(Rc<Vec<JValue>>),       // Rc-wrapped for O(1) clone
    Object(Rc<IndexMap<String, JValue>>),  // Rc-wrapped
    Undefined,                    // JSONata undefined value
    Lambda(Rc<StoredLambda>),     // First-class function carrying its closure
    Builtin {                     // First-class built-in function
        name: String,
    },
    Regex {                       // First-class regex value
        pattern: String,
        flags: String,
    },
}

Key Design Decisions¶

1. Rc Wrapping for Zero-Copy Cloning

// O(1) clone - just increments reference count
let s1 = JValue::string("hello");
let s2 = s1.clone();  // No data copy

// Compare to String clone: O(n)
let s1 = String::from("hello");
let s2 = s1.clone();  // Copies entire string

Impact: 20-100x performance improvement on realistic workloads.

2. First-Class Lambda/Builtin/Regex

Instead of wrapping in JSON objects:

// First-class variant (fast)
JValue::Lambda(Rc::new(stored_lambda))

// Tagged JSON object (slow)
JValue::Object(map! {
    "type" => "lambda",
    "id" => 1,
})

Impact: Enum discriminant check vs hash map lookup — and the lambda value IS the closure (body, params, signature, captured environment), so invoking a function value never needs a lookup at all.

3. Undefined as Distinct Type

// JSONata undefined != null
JValue::Undefined  // No value (not an error)
JValue::Null       // Explicit null value

Type Conversions¶

Python → JValue:

fn from_py(py: Python, obj: &PyAny) -> PyResult<JValue> {
    if obj.is_none() {
        Ok(JValue::Null)
    } else if let Ok(b) = obj.extract::<bool>() {
        Ok(JValue::Bool(b))
    } else if let Ok(n) = obj.extract::<f64>() {
        Ok(JValue::Number(n))
    // ...
}

JValue → Python:

fn to_py(&self, py: Python) -> PyResult<PyObject> {
    match self {
        JValue::Null => Ok(py.None()),
        JValue::Bool(b) => Ok(b.to_object(py)),
        JValue::Number(n) => Ok(n.to_object(py)),
        JValue::String(s) => Ok(s.to_object(py)),
        // ...
    }
}

Evaluation Pipeline¶

Expression Compilation¶

JSONata String  →  Tokenizer  →  Parser  →  AST
"items[price>100]"              →  Node tree

Expression Evaluation¶

AST  →  Evaluator  →  Context  →  Result
     ↓
  Functions
     ↓
  Scope Stack

Evaluation Steps¶

Parse expression to AST (once)
Create evaluation context with data
Traverse AST nodes recursively
Evaluate each node with context
Return result value

Example: Path Expression¶

// Expression: "items[price > 100].name"
// AST: Path(Identifier("items"),
//           Predicate(Comparison(field("price"), ">", Number(100))),
//           Identifier("name"))

fn evaluate_path(ctx: &mut Context, path: &PathNode) -> Result<JValue> {
    // 1. Evaluate base: items
    let base = evaluate_identifier(ctx, "items")?;

    // 2. Apply predicate: [price > 100]
    let filtered = apply_predicate(base, predicate)?;

    // 3. Extract field: .name
    let result = extract_field(filtered, "name")?;

    Ok(result)
}

Scope Stack and Lambda Storage¶

Context Structure¶

pub struct Context {
    scope_stack: Vec<Scope>,     // Stack of scopes
    parent_data: Option<JValue>,
}

pub struct Scope {
    bindings: HashMap<String, JValue>,  // Variable bindings
}

pub struct StoredLambda {
    params: Vec<String>,
    body: AstNode,
    signature: Option<String>,
    captured_env: HashMap<String, JValue>,  // Free variables, snapshot at definition
    captured_data: Option<JValue>,          // Data context at definition
    thunk: bool,                            // Body has optimizable tail calls
    self_name: Option<String>,              // Set for `$f := function ...` (letrec)
    // ...
}

Scope Management¶

Push/pop pattern (not clone/restore):

// Efficient - push/pop
ctx.push_scope();
let result = evaluate_expression(ctx, expr)?;
ctx.pop_scope();

// Inefficient - clone/restore
let old_ctx = ctx.clone();
let result = evaluate_expression(ctx, expr)?;
*ctx = old_ctx;

Lambda Storage¶

A lambda value carries its closure (issue #157): JValue::Lambda(Rc<StoredLambda>). The Rc provides the lifetime — a closure escaping its defining scope (returned from a block, nested in an array/object, captured by another closure) stays alive for as long as any value references it. There is no side table and no escape analysis; dangling lambda references are unrepresentable.

// Create lambda: the value IS the closure
let lambda_value = JValue::Lambda(Rc::new(StoredLambda {
    params: vec!["x".into()],
    body: body_ast,
    captured_env: capture_free_vars(ctx, &body_ast),
    // ...
}));

// Call lambda: no lookup — the value carries everything
if let JValue::Lambda(stored) = &callable {
    ctx.push_scope();
    // bind captured_env, then self_name (if any), then params
    let result = evaluate(ctx, &stored.body)?;
    ctx.pop_scope();
}

Recursion: Late-Bound Letrec¶

$f := function($n){ $n = 0 ? 0 : $f($n - 1) } must call itself even after $f has escaped its defining scope. The closure does not store a reference to itself (that would be an Rc cycle, i.e. a leak); instead it records its own self_name at definition, and each invocation binds that name to the closure being invoked. Free-variable capture is a by-value snapshot at definition time (not jsonata-js's live frames); self_name binds after captured_env, so a rebinding $f := function(){ ...$f()... } recurses into itself rather than the captured previous $f.

Selective Capture¶

Only capture free variables:

fn capture_free_vars(ctx: &Context, body: &AstNode) -> HashMap<String, JValue> {
    let free_vars = find_free_variables(body);
    let mut captured = HashMap::new();

    for var in free_vars {
        if let Some(value) = ctx.lookup(&var) {
            captured.insert(var, value.clone());
        }
    }

    captured
}

Performance Optimizations¶

Applied Optimizations¶

Rc-wrapped JValue - O(1) clone (20-100x speedup)
First-class Lambda/Builtin/Regex - Enum variants vs hash maps
Scope stack push/pop - Instead of HashMap clone
Selective lambda capture - Only capture free variables
Zero-copy field extraction - Reference-based access
Predicate short-circuit - Skip numeric evaluation for booleans
Specialized predicates - Optimize simple comparisons
Merge sort - O(n log n) sorting algorithm
Iterator-based aggregation - Zero-clone $sum/$max/$min
HOF selective args - Only pass needed arguments

Predicate Optimization¶

Simple field comparisons are optimized:

// Optimized path
items[price > 100]      // Direct field comparison
items[category = "A"]   // Direct field equality

// General path (not optimized)
items[$contains(name, "widget")]  // Function call

Memory Efficiency¶

Rc sharing prevents unnecessary copies:

let data = JValue::array(vec![...]);  // Large array

// Rc increment only - no data copy
let filtered = filter_array(&data, predicate)?;
let mapped = map_array(&filtered, mapper)?;
let sorted = sort_array(&mapped, comparator)?;

Key Design Patterns¶

1. Mirror JavaScript Structure¶

Code organization mirrors jsonata-js for maintainability:

JavaScript              Rust
----------              ----
parser.js        →      parser.rs
jsonata.js       →      evaluator.rs
functions.js     →      functions.rs
datetime.js      →      datetime.rs
signature.rs     →      signature.rs

2. Pattern Matching for Type Safety¶

match value {
    JValue::Number(n) => Ok(n * 2.0),
    JValue::String(s) => parse_number(&s),
    JValue::Array(arr) => sum_array(&arr),
    _ => Err(EvalError::TypeError("Expected number".into())),
}

3. Result-Based Error Handling¶

pub fn evaluate(ctx: &mut Context, node: &AstNode) -> Result<JValue, EvalError> {
    match node {
        AstNode::Number(n) => Ok(JValue::Number(*n)),
        AstNode::Path(path) => evaluate_path(ctx, path),
        // ...
    }
}

4. Context Threading¶

// Context passed through call chain
fn evaluate(ctx: &mut Context, node: &AstNode) -> Result<JValue> {
    let result = evaluate_child(ctx, child_node)?;
    // ...
}

fn evaluate_child(ctx: &mut Context, node: &AstNode) -> Result<JValue> {
    // Same context, can access bindings/lambdas
}

5. Lazy Evaluation¶

// Only evaluate branches that are needed
fn evaluate_conditional(ctx: &mut Context, cond: &AstNode,
                       then_branch: &AstNode,
                       else_branch: &AstNode) -> Result<JValue> {
    let cond_value = evaluate(ctx, cond)?;

    if is_truthy(&cond_value) {
        evaluate(ctx, then_branch)  // Only evaluate then
    } else {
        evaluate(ctx, else_branch)  // Only evaluate else
    }
}

Testing Strategy¶

Three-Layer Testing¶

Rust Unit Tests (31 tests)
Test individual functions
Located in src/*.rs with #[cfg(test)]
Python Integration Tests (~50 tests)
Test Python API
Located in tests/python/
Reference Test Suite (1258 tests)
Official JSONata compatibility tests
100% pass rate

Test Coverage¶

Target: 100% coverage (matching upstream jsonata-js)