What's Next & Going Further

You have built a complete interpreted programming language. This chapter looks at the features worth adding next and where to go from here.

Features to Add Next

1. Classes and Objects (OOP)

Add object-oriented programming to Lux:

class Counter {
    init(start) {
        this.count = start;
    }
    
    increment() {
        this.count = this.count + 1;
        return this.count;
    }
    
    get() {
        return this.count;
    }
}

let counter = Counter(0);
print counter.increment();  // 1
print counter.increment();  // 2
print counter.get();         // 2

Implementation approach:

• Add class keyword
• Add this keyword for self-reference
• Add method dispatch
• Add inheritance with extends
• Add super for parent class access

2. Module/Import System

Split code across files:

// math.lux
fn add(a, b) { return a + b; }
fn multiply(a, b) { return a * b; }

export { add, multiply };

// main.lux
import { add, multiply } from "math.lux";

print add(2, 3);       // 5
print multiply(4, 5);  // 20

Implementation approach:

• Add import and export keywords
• File path resolution
• Module caching (don't re-execute imports)
• Circular import detection

3. Pattern Matching

More powerful conditionals:

match value {
    0 -> print "zero",
    1 -> print "one",
    2..10 -> print "small",
    _ -> print "other"
}

// Match on types
match x {
    Number(n) if n > 0 -> print "positive number",
    String(s) -> print "string: " + s,
    List(items) -> print "list of " + str(len(items)),
    _ -> print "other"
}

Implementation approach:

• Add match keyword
• Pattern matching compiler
• Guard conditions
• Exhaustiveness checking

4. Error Handling in Lux

Handle errors gracefully:

// Try-catch style
try {
    let result = risky_operation();
    print result;
} catch e {
    print "Error: " + e;
}

// Result type (Rust-style)
fn divide(a, b) {
    if b == 0 {
        return Err("Division by zero");
    }
    return Ok(a / b);
}

match divide(10, 2) {
    Ok(value) -> print value,
    Err(msg) -> print "Error: " + msg
}

Implementation approach:

• Add try, catch, throw keywords
• Or add Result and Option types
• Unwrap operators (?)

5. Standard Library

Build a rich standard library:

// String utilities
import { split, join, trim, contains } from "std/string";

// Math functions
import { abs, sqrt, pow, sin, cos } from "std/math";

// File I/O
import { read_file, write_file } from "std/io";

let content = read_file("data.txt");
let lines = split(content, "\n");
for line in lines {
    print line;
}

// HTTP client
import { get, post } from "std/http";

let response = get("https://api.example.com/data");
print response.body;

Implementation as native functions:

• String: split, join, trim, upper, lower, replace
• Math: sqrt, pow, sin, cos, floor, ceil, round
• I/O: read_file, write_file, exists
• HTTP: get, post (with external crate)
• JSON: parse, stringify

6. Async/Await

Asynchronous programming:

async fn fetch_data(url) {
    let response = await http.get(url);
    return response.json();
}

async fn main() {
    let data1 = await fetch_data("https://api1.com");
    let data2 = await fetch_data("https://api2.com");
    print data1;
    print data2;
}

main();

Implementation:

• Add async and await keywords
• Event loop (tokio integration)
• Futures/Promises
• Concurrent execution

7. Iterators and Generators

Lazy evaluation:

// Generator
fn* range(start, end) {
    let i = start;
    while i < end {
        yield i;
        i = i + 1;
    }
}

for n in range(0, 10) {
    print n;
}

// Custom iterator
fn map(iter, f) {
    return fn*() {
        for item in iter {
            yield f(item);
        }
    };
}

Implementation:

• Add yield keyword
• Generator state machines
• Iterator protocol

8. String Interpolation

Embed expressions in strings:

let name = "Alice";
let age = 30;
print "My name is {name} and I am {age} years old";
// Output: My name is Alice and I am 30 years old

let x = 5;
let y = 10;
print "The sum of {x} and {y} is {x + y}";
// Output: The sum of 5 and 10 is 15

Implementation:

• Extend lexer to detect {...} in strings
• Parse expressions inside braces
• Generate concatenation AST

9. List and String Methods

Object-like method syntax:

let list = [1, 2, 3, 4, 5];
list.push(6);
list.pop();
let doubled = list.map(fn(x) { return x * 2; });
let evens = list.filter(fn(x) { return x % 2 == 0; });

let text = "hello world";
let upper = text.upper();
let words = text.split(" ");
print words[0];  // "hello"

Implementation:

• Add method call syntax: object.method(args)
• Built-in methods for each type
• User-defined methods (with prototypes or classes)

10. Break and Continue

Loop control flow:

let i = 0;
while true {
    if i >= 10 {
        break;  // Exit loop
    }
    if i % 2 == 0 {
        i = i + 1;
        continue;  // Skip to next iteration
    }
    print i;
    i = i + 1;
}

Implementation:

• Add break and continue keywords
• Use control flow exceptions
• Or add break/continue flags to loop evaluation

Performance Improvements

1. Bytecode Compilation

Instead of walking the AST, compile to bytecode:

Source → Lexer → Parser → AST → Compiler → Bytecode → VM

Advantages:

• 10-50x faster than tree-walking
• Smaller memory footprint
• Better optimization opportunities

Bytecode instructions:

LOAD_CONST 5        // Push 5 onto stack
LOAD_CONST 3        // Push 3 onto stack
ADD                 // Pop two, push sum
STORE_VAR "x"       // Store to variable x
LOAD_VAR "x"        // Load variable x
PRINT               // Pop and print

2. Stack-Based VM

Implement a stack machine:

pub struct VM {
    stack: Vec<Value>,
    globals: HashMap<String, Value>,
    frames: Vec<CallFrame>,
}

impl VM {
    pub fn execute(&mut self, chunk: &Chunk) -> Result<(), String> {
        for instruction in &chunk.code {
            match instruction {
                Instruction::LoadConst(value) => {
                    self.stack.push(value.clone());
                }
                Instruction::Add => {
                    let b = self.stack.pop().unwrap();
                    let a = self.stack.pop().unwrap();
                    self.stack.push(self.add(a, b)?);
                }
                Instruction::Print => {
                    let value = self.stack.pop().unwrap();
                    println!("{}", value.to_string());
                }
                // ... more instructions
            }
        }
        Ok(())
    }
}

3. Garbage Collection

Implement automatic memory management:

Mark and Sweep:

1. Mark phase: Traverse reachable objects
2. Sweep phase: Free unmarked objects

struct GC {
    objects: Vec<Rc<RefCell<Object>>>,
    roots: Vec<Rc<RefCell<Object>>>,
}

impl GC {
    fn collect(&mut self) {
        self.mark();
        self.sweep();
    }
    
    fn mark(&mut self) {
        for root in &self.roots {
            self.mark_object(root);
        }
    }
    
    fn sweep(&mut self) {
        self.objects.retain(|obj| obj.borrow().marked);
    }
}

Generational GC:

• Young generation: Frequently collected
• Old generation: Rarely collected
• Much better performance

4. Just-In-Time (JIT) Compilation

Compile hot code paths to machine code at runtime:

Bytecode → Profile → Identify hot code → Compile to native → Execute

Use LLVM or Cranelift for code generation.

5. Optimization Passes

Optimize the AST or bytecode:

• Constant folding: 2 + 3 → 5 at compile time
• Dead code elimination: Remove unreachable code
• Inline caching: Cache method lookups
• Loop unrolling: Duplicate loop bodies
• Tail call optimization: Eliminate recursion overhead

Comparison: Implementation Strategies

Complete Lux Grammar (BNF)

program        → declaration* EOF ;

declaration    → classDecl
               | funDecl
               | varDecl
               | statement ;

classDecl      → "class" IDENTIFIER ( "<" IDENTIFIER )?
                 "{" function* "}" ;
funDecl        → "fn" function ;
varDecl        → "let" IDENTIFIER ( "=" expression )? ";" ;

statement      → exprStmt
               | forStmt
               | ifStmt
               | printStmt
               | returnStmt
               | whileStmt
               | block ;

exprStmt       → expression ";" ;
forStmt        → "for" IDENTIFIER "in" expression block ;
ifStmt         → "if" expression block
                 ( "else" block )? ;
printStmt      → "print" expression ";" ;
returnStmt     → "return" expression? ";" ;
whileStmt      → "while" expression block ;
block          → "{" declaration* "}" ;

expression     → assignment ;

assignment     → ( call "." )? IDENTIFIER "=" assignment
               | logic_or ;

logic_or       → logic_and ( "or" logic_and )* ;
logic_and      → equality ( "and" equality )* ;
equality       → comparison ( ( "!=" | "==" ) comparison )* ;
comparison     → term ( ( ">" | ">=" | "<" | "<=" ) term )* ;
term           → factor ( ( "-" | "+" ) factor )* ;
factor         → unary ( ( "/" | "*" | "%" ) unary )* ;

unary          → ( "!" | "-" | "not" ) unary | call ;

call           → primary ( "(" arguments? ")" | "." IDENTIFIER | "[" expression "]" )* ;

primary        → "true" | "false" | "nil" | "this"
               | NUMBER | STRING | IDENTIFIER | "(" expression ")"
               | "[" arguments? "]"
               | "fn" "(" parameters? ")" block ;

function       → IDENTIFIER "(" parameters? ")" block ;
parameters     → IDENTIFIER ( "," IDENTIFIER )* ;
arguments      → expression ( "," expression )* ;

NUMBER         → DIGIT+ ( "." DIGIT+ )? ;
STRING         → '"' <any char except '"'>* '"' ;
IDENTIFIER     → ALPHA ( ALPHA | DIGIT )* ;
ALPHA          → 'a' ... 'z' | 'A' ... 'Z' | '_' ;
DIGIT          → '0' ... '9' ;

Resources for Further Learning

Books

1. Crafting Interpreters by Robert Nystrom

   • The definitive guide to building interpreters
   • Covers tree-walking and bytecode VMs
   • Available free online: https://craftinginterpreters.com/

2. Writing An Interpreter In Go by Thorsten Ball

   • Build an interpreter from scratch
   • Excellent step-by-step approach
   • Focus on Go, but concepts apply to any language

3. Engineering a Compiler by Keith Cooper and Linda Torczon

   • Comprehensive compiler engineering textbook
   • Advanced optimizations
   • Industry-standard reference

4. Programming Language Pragmatics by Michael Scott

   • Broad overview of language design
   • Covers interpretation, compilation, and VMs
   • Great for understanding design trade-offs

5. Language Implementation Patterns by Terence Parr

   • Practical patterns for parsers, interpreters, and compilers
   • Includes ANTLR examples

Online Courses

• Stanford CS143: Compilers
• Coursera: Compilers (Stanford)
• MIT 6.035: Computer Language Engineering
• MOOC: Programming Languages (University of Washington)

Tools and Libraries

• LLVM: Compiler infrastructure for code generation
• Cranelift: Fast code generator
• ANTLR: Parser generator
• Tree-sitter: Incremental parsing library
• Roslyn: C# compiler as a service

Communities

• r/ProgrammingLanguages on Reddit
• Language Design Discord
• LLVM Discourse
• Lambda the Ultimate (blog)

Real-World Language Projects

Study these to see production implementations:

Your Lux Implementation Recap

Here is what you have built:

• Lexical Analysis: tokenization with position tracking
• Syntax Analysis: recursive descent parser with precedence
• Semantic Analysis: type checking at runtime
• Interpretation: tree-walking evaluator
• Memory Management: Rust's ownership plus Rc/RefCell
• Scoping: lexical scoping with closures
• Functions: first-class with closure capture
• Control Flow: if, while, for, return
• Data Structures: lists and strings
• Error Handling: comprehensive error reporting
• REPL: interactive development environment
• Testing: unit and integration tests

How to Add a Feature

1. Pick a feature from the list above.
2. Design the syntax: how will it look in Lux?
3. Update the lexer: add new tokens if needed.
4. Update the parser: parse the new syntax.
5. Update the interpreter: execute the new feature.
6. Write tests: confirm it works correctly.
7. Update documentation: document the feature.
8. Iterate: refine based on usage.

Final Thoughts

Building a programming language teaches you:

• How languages work under the hood
• Data structure design
• Algorithm implementation
• Software architecture
• Testing strategies
• API design

These skills transfer to:

• Building DSLs for your domain
• Understanding other languages better
• Debugging complex systems
• Designing better APIs
• System architecture

You are now equipped to:

• Build custom scripting languages
• Create domain-specific languages (DSLs)
• Contribute to existing language projects
• Design language features
• Implement compilers and interpreters professionally

Summary

Where to Go From Here

You have a working interpreter and a long list of features to choose from. Pick one and build it. The best way to deepen your understanding of language implementation is to keep extending Lux, or to read another implementation end to end. A few good next steps:

• Read Crafting Interpreters chapters 14 to 30 to build a bytecode VM in C and compare the design choices.
• Pick one feature from the list above (classes, modules, pattern matching) and add it to Lux end to end: lexer, parser, interpreter, tests.
• Study a small production interpreter like Wren or Lox to see how design choices play out in a real codebase.
• Build a small DSL on top of Lux for a domain you actually work in. That is where these skills pay off.