The History of JSONL
From humble beginnings to universal adoption - discover how JSON Lines became the standard for streaming structured data across the modern web.
The Pre-JSONL Era (1990s-2009)
The challenges that led to the creation of JSON Lines format.
The Data Streaming Problem
Before JSONL, developers faced significant challenges when working with streaming JSON data. Traditional JSON arrays required complete parsing before any processing could begin, creating bottlenecks in data pipelines.
Memory Constraints
Loading multi-gigabyte JSON arrays into memory was impractical and often impossible on production servers.
Slow Processing
Waiting for entire datasets to load before processing meant delayed insights and slow pipelines.
Format Fragmentation
Each organization created custom delimited formats, leading to incompatibility and vendor lock-in.
Existing Solutions and Their Limitations
CSV (Comma-Separated Values)
The de facto standard for tabular data since the 1970s.
Pros:
- Simple and widely supported
- Human-readable
- Streamable line-by-line
Cons:
- No nested structures
- No data types (everything is text)
- Escaping issues with delimiters
XML (Extensible Markup Language)
Dominant structured data format in the 1990s and early 2000s.
Pros:
- Nested structures
- Schema validation (XSD)
- Mature tooling
Cons:
- Verbose and bloated
- Difficult to stream
- Complex parsing requirements
JSON Arrays
The emerging standard for web APIs in the late 2000s.
Pros:
- Lightweight syntax
- Native JavaScript support
- Nested structures
Cons:
- Not streamable
- Requires complete parse
- Memory intensive for large datasets
The JSON Revolution (2001-2009)
2001
Douglas Crockford specifies JSON format, derived from JavaScript object literal syntax.
2005-2006
AJAX (Asynchronous JavaScript and XML) popularizes JSON for web APIs, gradually replacing XML.
2007-2009
JSON becomes the dominant format for web APIs. Companies struggle with large JSON datasets in log processing and data pipelines.
By 2009, JSON had won the data interchange format war for web APIs, but the question remained: how do we efficiently stream large collections of JSON objects?
Birth of JSON Lines (2010-2012)
The emergence of a simple yet powerful solution to the streaming JSON problem.
The Brilliant Simplicity
Around 2010-2011, multiple developers independently arrived at the same elegant solution: what if we simply put one JSON object per line?
The Core Principle
Each line is valid JSON. Each line is independent. Process line-by-line. That's it.
A Format by Many Names
The format emerged organically across different communities, each giving it their own name.
JSON Lines (JSONL)
The name that would eventually become the standard. Simple, descriptive, and self-explanatory.
Used by: Early adopters, documentation sites, general community
Newline-Delimited JSON (NDJSON)
Emphasized the delimiter mechanism. More technically precise terminology.
Used by: Data engineering communities, ETL tool developers
Line-Delimited JSON (LDJSON)
Similar to NDJSON, less commonly used variant.
Used by: Some database vendors, scattered tooling
JSON Streaming / JSON Sequence
Used informally before standardization, often confused with other streaming JSON approaches.
Used by: Early blog posts, internal documentation
While "JSON Lines" and "NDJSON" are often used interchangeably today, they refer to the same format. JSONL became the more popular shorthand.
Early Adopters (2011-2013)
Log Processing Systems
The first major use case. Log aggregation systems needed to process millions of log entries per second without loading everything into memory.
Tools like Logstash (later part of the Elastic Stack) embraced JSONL for log shipping and processing pipelines.
Data Science Community
Python data scientists working with large datasets found JSONL perfect for streaming data processing.
The format worked seamlessly with Python's line-by-line file reading patterns, making it a natural fit.
Database Import/Export Tools
NoSQL databases like MongoDB needed efficient bulk import/export formats.
MongoDB's mongoimport and mongoexport tools added JSONL support, allowing for efficient data migration.
Formalization & Standardization (2013-2015)
The community comes together to document and standardize the format.
The Need for Specification
As adoption grew, inconsistencies emerged. Some implementations used different line endings (LF vs CRLF), others had different rules about empty lines or whitespace. The community needed a clear specification.
UTF-8 Encoding
Mandate UTF-8 as the standard character encoding, ensuring international compatibility.
Line Separator Clarity
Define LF (\n) or CRLF (\r\n) as acceptable line separators, accommodating different platforms.
No BOM (Byte Order Mark)
Explicitly disallow BOM to prevent parsing issues and maintain clean line-by-line processing.
One JSON Per Line
Clarify that each line must contain exactly one valid JSON value (typically an object).
Community Documentation Efforts
jsonlines.org (2013)
The community-created website that became the de facto documentation source. Provided clear examples and explained the format's benefits.
ndjson.org
Parallel documentation effort under the NDJSON name, essentially describing the same format with slightly different terminology.
GitHub Specifications
Various GitHub repositories provided reference implementations and test suites, helping implementations stay consistent.
Unlike JSON (which went through formal ECMA standardization), JSONL remained a community-driven format. This flexibility allowed rapid evolution and adoption.
RFC Discussions and Related Standards
RFC 7464 - JSON Text Sequences (2015)
The IETF published RFC 7464, describing a similar but distinct format using the RS (Record Separator, ASCII 0x1E) character.
Key Difference:
RFC 7464 uses <RS>JSON<LF> format, while JSONL uses simple JSON<LF>. The RFC format never gained widespread adoption due to the RS character being less standard-tool friendly.
Why JSONL Won
- Simpler format - works with standard Unix tools (grep, sed, awk)
- Already had widespread adoption before RFC 7464 was published
- Human-readable line separators (visible newlines vs invisible RS character)
- Compatible with existing text editors and version control systems
Widespread Adoption (2015-2020)
JSONL becomes the standard across industries and platforms.
Cloud Platform Adoption
Amazon Web Services (AWS)
AWS adopted JSONL across multiple services:
- Amazon Athena: JSONL as a native format for querying S3 data
- AWS Kinesis: Stream processing with JSONL records
- CloudWatch Logs: JSONL export format for log analysis
- Amazon SageMaker: JSONL for training data in machine learning
Google Cloud Platform (GCP)
Google embraced JSONL for big data:
- BigQuery: JSONL as primary import/export format
- Cloud Storage: Optimized handling for JSONL files
- Dataflow: Native JSONL support in streaming pipelines
- Cloud Logging: JSONL export for log analytics
Microsoft Azure
Azure integrated JSONL across analytics:
- Azure Data Lake: JSONL storage and querying
- Azure Stream Analytics: Real-time JSONL processing
- Azure Monitor: JSONL log exports
Database System Integration
MongoDB
mongoimport/mongoexport utilities with native JSONL support for bulk operations
PostgreSQL
COPY command with JSONL format support for efficient data loading
Elasticsearch
Bulk API using JSONL for high-performance indexing
Apache Kafka
JSONL as standard format for message serialization
ClickHouse
JSONEachRow format (JSONL) for fast analytics ingestion
Redis
Redis Modules using JSONL for bulk data operations
Programming Language Libraries
By 2018, every major programming language had mature JSONL libraries.
Python
jsonlines, pandas, ndjson packages
JavaScript
ndjson, split2, JSONStream
Go
encoding/json with bufio
Java
Jackson, Gson with streaming
Ruby
jsonl gem
Rust
serde_json with BufReader
Industry Sector Adoption
Machine Learning & AI
JSONL became standard for training datasets, model outputs, and MLOps pipelines. Hugging Face, OpenAI, and others standardized on JSONL for dataset distribution.
Analytics & Business Intelligence
Data warehouses and BI tools adopted JSONL for ETL processes, making semi-structured data analysis more accessible.
Observability & Monitoring
Log aggregation platforms (Splunk, Datadog, New Relic) embraced JSONL for structured logging and metrics ingestion.
DevOps & CI/CD
Build logs, deployment records, and infrastructure events standardized on JSONL for parsing and analysis.
Modern Era: The First AI Wave (2020-2022)
Large-scale model training turns JSONL into the default shape of a dataset.
The AI & LLM Revolution (2020-2022)
Large Language Models and modern AI systems dramatically increased JSONL adoption. Training datasets containing billions of examples needed efficient storage and streaming, and JSONL was the perfect fit.
Pretraining Corpora
The web-scale text collections behind this generation of models were assembled, filtered, and deduplicated as JSONL, because every stage of that pipeline reads one document and writes it back out.
Hugging Face and Open Weights
The leading model hub distributed thousands of datasets as JSONL and built its streaming datasets API around reading them a record at a time. The open-weight ecosystem inherited the same conventions.
What happened next, from 2023 onward, turned a popular convention into the connective tissue of an entire industry.
The LLM Toolchain Era (2023-2026)
Fine-tuning, batch inference, evals, observability, and query engines that finally read JSONL in place.
The Universal Fine-Tuning Interchange Format
Between 2023 and 2026 every major model provider settled on the same file format for fine-tuning data, and nobody standardized it. JSONL had been the container since examples were prompt-and-completion pairs; the chat models added the shape of each line, an array of role-tagged messages. OpenAI's guide, for the endpoint it introduced in August 2023, asks for JSONL holding a messages array and links the word JSONL out to jsonlines.org, because there is no spec to link to.
Same Container, Different Vocabularies
Convergence was at the file level, not the schema level:
- Google Vertex AI: JSONL, but a
contentsarray ofparts, with the assistant role spelled model - Mistral: its fine-tuning API, launched June 2024, used OpenAI's naming
- TRL and axolotl: TRL calls the shape conversational; axolotl made the field name configurable, because nobody agreed on one
Why JSONL Won It
No committee picked it. JSONL is appendable, so new examples are just new lines; streamable, so training reads one record at a time from a file larger than memory; diff-friendly, so a dataset reviews like source code; and emittable by any language that can write a newline. More on the advantages of JSONL.
Batch Inference: One Request Per Line
The request envelope converged too. OpenAI launched its Batch API in April 2024: upload a JSONL file whose every line carries a correlation id, a method, a URL, and a body, then download a JSONL file whose every line carries that id and the response. Mistral shipped an equivalent in November 2024, and Bedrock, Azure OpenAI, Groq, and Together landed on the same shape. Anthropic posts a JSON array but returns JSONL.
The id is not decoration. Anthropic, AWS, Azure, and Together all document that output line order is not guaranteed to match input order. JSONL gives you a record per line, not a sequence.
Evals, Retrieval Manifests, and Agent Traces
EleutherAI's lm-evaluation-harness writes per-sample logs as JSONL while the aggregate score summary sits beside it as ordinary JSON: a summary is one object, a log is an open-ended sequence. Benchmarks followed, with HumanEval, GSM8K, MBPP, and HellaSwag all shipping as JSONL, though MMLU was released as CSV.
Retrieval pipelines adopted JSONL for chunk manifests, one line per chunk, so re-chunking a document rewrites only that document's lines. Agent transcripts followed, since a session is an append-only sequence of events. Claude Code stores session transcripts as JSONL, then warns that the per-line schema is internal and changes between releases: the container is agreed, the contents are not.
Query Engines Made JSONL Directly Readable
For everyday data work this was the biggest practical shift since 2020. DuckDB 0.7.0, released in February 2023, shipped a newline-delimited JSON reader, so one self-contained binary could run SQL directly against a JSONL file on disk or in object storage. Polars answered with a lazy newline-delimited scanner that pushes filters and column projections into the scan.
What this removed was the conversion toll. Exploring a JSONL export used to begin with converting it to a columnar format. By the mid-2020s that became a decision you make once the data proves worth keeping, not a fee to look at it.
Observability Standardized on Newline-Delimited JSON
The telemetry pipeline rebuilt itself, and the replacement agents all speak newline-delimited JSON. Vector pairs a JSON codec with newline-delimited framing on its file and HTTP sinks, Fluent Bit offers a JSON-lines format, and the OpenTelemetry file exporter specification calls its output a JSON lines file and names the JSONL extension as preferred.
Fluentd is not gone, and remains a graduated CNCF project, but the center of gravity moved: AWS put Fluentd support for Container Insights into maintenance mode, Red Hat made Vector the default collector in OpenShift Logging 5.6, and in 2025 the CNCF published a migration guide to Fluent Bit.
The Standardization Gap That Never Closed
For all that adoption, the format is no more official in 2026 than in 2014. There is still no IANA-registered media type for JSON Lines, nor for the newline-delimited variant everyone sends, whose leading x marks it unregistered by construction, yet that is what Elasticsearch and OpenSearch bulk endpoints accept. One JSON sequence type is registered, RFC 7464 from 2015, but it prefixes every record with an ASCII Record Separator, so it is not JSONL. The standards-track answer existed and the industry declined it.
Two informal specs still sit side by side: jsonlines.org, created by Ian Ward in 2013 and still maintained, which concedes that its media type is not yet standardized; and the rival newline-delimited JSON spec, frozen at 1.0.0 since October 2014, with an open issue asking someone to register a type.
That is the real lesson of the last four years. JSONL was never ratified, never versioned, never governed. It won by being the smallest structure that still lets a machine read one record at a time. Its actual rules are on the JSONL definition page.
Current State of JSONL
Universal Adoption
- Supported by all major cloud platforms
- Default for ML/AI datasets
Ecosystem Maturity
- Libraries in every major language
- Built-in support in Unix tools (jq)
- IDE extensions and syntax highlighting
Future Directions
Schema Evolution
Conventions for schema versioning and validation within JSONL files, combining flexibility with structure.
A Registered Media Type
The unfinished business the format has carried since 2014, and the last thing standing between JSONL and full first-class status.
JSONL Timeline at a Glance
Streaming JSON problem identified
Format emerges organically, early adopters
Community documentation, specifications published
Widespread adoption - cloud platforms, databases, tools
AI/ML boom - LLM training corpora, dataset hubs, modern data systems
LLM toolchain - fine-tuning files, batch APIs, evals, agent traces, in-place querying
Why JSONL Succeeded
Simplicity
One JSON object per line. That's the entire specification. Simple enough to implement in minutes, powerful enough for petabytes of data.
Tool Compatibility
Works with standard Unix tools (grep, sed, awk), text editors, version control, and every JSON parser ever written.
Performance
Stream processing without loading entire datasets into memory. Process millions of records per second on commodity hardware.
Explore More About JSONL
Dive deeper into the format that powers modern data infrastructure.