Common JSONL Mistakes

Jump to a Mistake

JSON requires double quotes for both keys and string values. Single quotes are valid in JavaScript and Python, but they are not valid JSON. This mistake often happens when developers copy output from a Python REPL or JavaScript console.

{'name': 'Alice', 'age': 30}
{'name': 'Bob', 'age': 25}

{"name": "Alice", "age": 30}
{"name": "Bob", "age": 25}

In JavaScript, object keys do not need quotes if they are valid identifiers. In JSON, all keys must be double-quoted strings. This is a strict requirement of the JSON specification.

{name: "Alice", age: 30}
{name: "Bob", active: true}

{"name": "Alice", "age": 30}
{"name": "Bob", "active": true}

Every record must be separated by a newline character (\n). When multiple JSON objects are concatenated on the same line, parsers will fail because the combined text is not valid JSON.

{"name": "Alice", "age": 30}{"name": "Bob", "age": 25}{"name": "Charlie", "age": 35}

{"name": "Alice", "age": 30}
{"name": "Bob", "age": 25}
{"name": "Charlie", "age": 35}

The JSON specification (RFC 8259) requires UTF-8 encoding. Files saved in Latin-1, Windows-1252, Shift-JIS, or other encodings will cause parse errors whenever a non-ASCII character appears. Characters like accented letters, currency symbols, or emoji will be corrupted or rejected.

# Latin-1 byte for e-acute: E9
# Parser expects UTF-8 byte: C3 A9

{"name": "René", "city": "Montréal"}
# Fails with: invalid UTF-8 byte sequence

# Two valid options for special characters:

# Option 1: Raw UTF-8 characters
{"name": "René", "city": "Montréal"}

# Option 2: Unicode escape sequences
{"name": "Ren\u00e9", "city": "Montr\u00e9al"}

While the JSONL specification says parsers may ignore empty lines, many real-world parsers do not. An empty line between records can cause parse failures, off-by-one errors in line counts, or unexpected null values. It is safest to avoid them entirely.

{"name": "Alice", "age": 30}

{"name": "Bob", "age": 25}

{"name": "Charlie", "age": 35}

{"name": "Alice", "age": 30}
{"name": "Bob", "age": 25}
{"name": "Charlie", "age": 35}

Programmatic Validation

Catch all 10 mistakes automatically with code. Here are ready-to-use validation snippets in Python, Node.js, and jq.

import json
import sys

def validate_jsonl(filepath):
    """Validate a JSONL file and report errors by line."""
    errors = []
    valid = 0

    with open(filepath, 'r', encoding='utf-8-sig') as f:
        for line_num, line in enumerate(f, 1):
            # Skip empty lines (but warn about them)
            if not line.strip():
                errors.append(f"Line {line_num}: Empty line (remove it)")
                continue

            try:
                obj = json.loads(line)
                valid += 1
            except json.JSONDecodeError as e:
                errors.append(f"Line {line_num}: {e}")

    # Print results
    print(f"Valid records: {valid}")
    print(f"Errors: {len(errors)}")

    for err in errors:
        print(f"  {err}")

    return len(errors) == 0

# Usage
if __name__ == '__main__':
    filepath = sys.argv[1] if len(sys.argv) > 1 else 'data.jsonl'
    is_valid = validate_jsonl(filepath)
    sys.exit(0 if is_valid else 1)

const fs = require('fs');
const readline = require('readline');

async function validateJsonl(filepath) {
    const fileStream = fs.createReadStream(filepath, { encoding: 'utf-8' });
    const rl = readline.createInterface({
        input: fileStream,
        crlfDelay: Infinity
    });

    let lineNum = 0;
    let valid = 0;
    const errors = [];

    for await (const line of rl) {
        lineNum++;

        // Strip BOM from first line if present
        const cleanLine = lineNum === 1
            ? line.replace(/^\uFEFF/, '')
            : line;

        // Skip empty lines (but warn)
        if (cleanLine.trim() === '') {
            errors.push(`Line ${lineNum}: Empty line (remove it)`);
            continue;
        }

        try {
            JSON.parse(cleanLine);
            valid++;
        } catch (e) {
            errors.push(`Line ${lineNum}: ${e.message}`);
        }
    }

    console.log(`Valid records: ${valid}`);
    console.log(`Errors: ${errors.length}`);
    errors.forEach(err => console.log(`  ${err}`));

    return errors.length === 0;
}

// Usage
const filepath = process.argv[2] || 'data.jsonl';
validateJsonl(filepath).then(isValid => {
    process.exit(isValid ? 0 : 1);
});

# Quick validation - check if every line is valid JSON
jq -e . data.jsonl > /dev/null 2>&1 && echo "Valid JSONL" || echo "Invalid JSONL"

# Show line numbers of invalid lines
awk '{print NR": "$0}' data.jsonl | while IFS= read -r line; do
    num=$(echo "$line" | cut -d: -f1)
    content=$(echo "$line" | cut -d: -f2-)
    echo "$content" | jq . > /dev/null 2>&1 || echo "Error on line $num"
done

# Count valid vs total lines (excluding empty lines)
total=$(grep -c '.' data.jsonl)
valid=$(jq -c . data.jsonl 2>/dev/null | wc -l)
echo "Valid: $valid / $total"

# Remove empty lines and re-validate
sed '/^$/d' data.jsonl | jq -c . > cleaned.jsonl