JSONL Performance

Python: JSON vs JSONL Parsing

Recorded environment: 100,000 records, ~50MB file, Python 3.11, MacBook Pro M1. Python 3.11 is the interpreter these figures were measured on, not a recommendation - they have not been re-measured on Python 3.14.

# Benchmark code
import json
import orjson
import gzip
import time
import tracemalloc

def benchmark_json_array():
    tracemalloc.start()
    start = time.time()
    with open('data.json', 'r') as f:
        data = json.load(f)  # Load entire array
        for record in data:
            process(record)
    elapsed = time.time() - start
    peak = tracemalloc.get_traced_memory()[1] / 1024 / 1024
    tracemalloc.stop()
    print(f"JSON Array: {elapsed*1000:.0f}ms, {peak:.0f}MB")

def benchmark_jsonl_orjson():
    tracemalloc.start()
    start = time.time()
    with open('data.jsonl', 'rb') as f:
        for line in f:
            record = orjson.loads(line)
            process(record)
    elapsed = time.time() - start
    peak = tracemalloc.get_traced_memory()[1] / 1024 / 1024
    tracemalloc.stop()
    print(f"JSONL (orjson): {elapsed*1000:.0f}ms, {peak:.0f}MB")

JavaScript/Node.js: Streaming Performance

Recorded environment: 100,000 records, ~50MB file, Node.js 20.x, MacBook Pro M1. Node.js 20.x is the runtime these figures were measured on, not a recommendation - they have not been re-measured on Node.js 24 LTS.

// Fast JSONL streaming with ndjson
const fs = require('fs');
const ndjson = require('ndjson');

const stream = fs.createReadStream('data.jsonl')
  .pipe(ndjson.parse())
  .on('data', (record) => {
    // Process each record as it arrives
    process(record);
  })
  .on('end', () => {
    console.log('Complete');
  });

Go: Concurrency Benchmark

Recorded environment: 1,000,000 records, ~500MB file, Go 1.21, MacBook Pro M1, 10 cores. Go 1.21 is the toolchain these figures were measured on, not a recommendation - they have not been re-measured on Go 1.26.

About 7.6x faster in this run: 3,200ms against 420ms across 10 goroutines. JSONL's line-based format is trivially parallelizable, since a worker can be handed any chunk of whole lines without parsing what came before it.

// Go: Parallel JSONL processing
package main

import (
    "bufio"
    "encoding/json"
    "os"
    "sync"
)

func processChunk(lines [][]byte, wg *sync.WaitGroup) {
    defer wg.Done()
    for _, line := range lines {
        var record map[string]interface{}
        json.Unmarshal(line, &record)
        // Process record...
    }
}

func main() {
    file, _ := os.Open("data.jsonl")
    defer file.Close()

    scanner := bufio.NewScanner(file)
    const chunkSize = 10000
    var chunk [][]byte
    var wg sync.WaitGroup

    for scanner.Scan() {
        chunk = append(chunk, append([]byte(nil), scanner.Bytes()...))

        if len(chunk) >= chunkSize {
            wg.Add(1)
            go processChunk(chunk, &wg)
            chunk = nil
        }
    }

    if len(chunk) > 0 {
        wg.Add(1)
        go processChunk(chunk, &wg)
    }

    wg.Wait()
}

Compression Ratios

JSONL compresses extremely well because every line repeats the same field names. The sizes below are a single measurement of one 100.0 MB JSONL file of flat, one-record-per-line user objects, each compressed once with the exact command shown:

gzip  -6 -k data.jsonl     # -> data.jsonl.gz
gzip  -9 -k data.jsonl     # maximum deflate level
bzip2 -9 -k data.jsonl     # -9 is bzip2's default
xz    -6 -k data.jsonl     # -6 is xz's default
zstd  -3 -k data.jsonl     # -3 is zstd's default

Streaming Compressed JSONL

Process compressed JSONL without decompressing entire file first:

import gzip
import json

# Stream-decompress and process
with gzip.open('data.jsonl.gz', 'rt') as f:
    for line in f:
        obj = json.loads(line)
        # Memory stays constant!

const fs = require('fs');
const zlib = require('zlib');
const readline = require('readline');

// readline is not a writable stream, so you cannot .pipe() into it.
// Hand the gunzip stream to createInterface as its "input" instead.
async function main() {
  const rl = readline.createInterface({
    input: fs.createReadStream('data.jsonl.gz').pipe(zlib.createGunzip()),
    crlfDelay: Infinity
  });

  for await (const line of rl) {
    if (line.trim() === '') continue;
    const obj = JSON.parse(line);
    // Process...
  }
}

main().catch(err => {
  console.error(err);
  process.exitCode = 1;
});

# Decompress and process on-the-fly with jq
zcat data.jsonl.gz | jq '.name'

# Decompress, filter, compress again
zcat input.jsonl.gz | grep '"status":"active"' | gzip > filtered.jsonl.gz

Performance impact: Decompression costs CPU time. In the Python table above - the only place on this page where both were actually timed on the same machine and file - adding gzip took the orjson run from 410ms to 680ms. Whether that is a net loss depends on your storage: reading roughly one eighth as many bytes usually more than pays for the CPU, and the slower the disk or network, the more compression wins.