How Real Platforms Use JSONL
Seven documented platforms - Elasticsearch, Apache Spark, ClickHouse, DuckDB, Hugging Face, OpenAI, and the major cloud data services - and exactly how each one reads, writes, or requires newline-delimited JSON.
How to Read This Page
These are not customer testimonials and they are not anonymized success stories. Every platform below is one you can install, sign up for, or read the source of, and every claim on this page is either a quote from that vendor's own public documentation or a link to it.
- Every number carries a source link. Where a vendor publishes a figure, it is quoted with the conditions the vendor stated - hardware, dataset size, and date included.
- Where no public figure exists, there is no figure. Behaviour is described instead. No estimated savings, no modelled percentages, no dollar amounts.
- The trade-offs are included. At least one platform here explicitly recommends against JSONL for its highest-throughput path, and that is documented below rather than omitted.
Elasticsearch Bulk API
NDJSON is the wire format, not an option
What the Documentation Says
Elasticsearch's _bulk endpoint does not accept a JSON array of documents. Elastic's API reference specifies that "the actions are specified in the request body using a newline delimited JSON (NDJSON) structure", following the repeating pattern of an action and metadata line, then an optional source line. This is the single highest-volume ingestion path into Elasticsearch, and it is defined in terms of JSONL.
Source: Elasticsearch API reference - Bulk index or delete documents
Constraints the Format Imposes
Choosing a line-delimited body has direct, documented consequences for how you build the request:
You Cannot Pretty-Print the Body
Literal newline characters are the delimiters, so every JSON document must be serialized onto exactly one line. Elastic's reference notes the same requirement drives the use of the curl --data-binary flag, because clients that normalize whitespace will corrupt the request.
The Last Line Must End With a Newline
The reference is explicit: "The final line of data must end with a newline character", and each newline may be preceded by a carriage return. A missing trailing newline is a common first-time failure.
One Thread Will Not Saturate a Cluster
Elastic's indexing-speed guidance states that "a single thread sending bulk requests is unlikely to be able to max out the indexing capacity of an Elasticsearch cluster", and advises pausing with randomized exponential backoff when the cluster signals rejection.
How the Bulk Body Is Structured
Documented Behaviour
- NDJSON structure: the body repeats an action and metadata line, then an optional source line, for index, create, update, and delete operations
- Content-Type: requests use a header of
application/jsonorapplication/x-ndjson - Independent line results: the response reports the outcome of each operation separately, so a single malformed document does not discard the batch
- Streamable: because record boundaries are newlines, the request can be produced and consumed incrementally rather than buffered whole
Example Bulk API Request (NDJSON)
{"index":{"_index":"logs","_id":"1"}}
{"timestamp":"2026-01-15T14:32:15Z","level":"ERROR","service":"api","message":"Database timeout"}
{"index":{"_index":"logs","_id":"2"}}
{"timestamp":"2026-01-15T14:32:16Z","level":"INFO","service":"web","message":"Request completed"}
{"index":{"_index":"logs","_id":"3"}}
{"timestamp":"2026-01-15T14:32:17Z","level":"WARN","service":"auth","message":"Rate limit exceeded"}Each pair of lines is one document: the action line, then the document source line.
What Elastic Actually Publishes About Speed
Elastic does not publish a headline speedup multiple for the Bulk API. What it publishes is a direction and a method, and both are worth more than a number you cannot reproduce:
"Bulk requests will yield much better performance than single-document index requests." Elastic states the direction and deliberately does not attach a multiplier, because the result depends on document size, hardware, and cluster shape.
"First try to index 100 documents at once, then 200, then 400, etc. doubling the number of documents in a bulk request in every benchmark run." The correct batch size is the one you measure, not one you copy.
"It is advisable to avoid going beyond a couple tens of megabytes per request even if larger requests seem to perform better" - larger bulk bodies raise memory pressure across the cluster.
Elastic recommends multiple threads or workers sending bulk requests concurrently, with randomized exponential backoff when the cluster starts rejecting.
Source: Elastic - Tune for indexing speed
Where This Path Gets Used
- Log shippers: Logstash, Filebeat, and Fluentd all deliver to Elasticsearch through the same
_bulkendpoint and therefore the same NDJSON body - Migrations and reindexing: ETL jobs emit JSONL and stream it straight into the bulk request
- Observability: traces, metrics, and spans arrive as bulk batches from APM agents
- Search index builds: catalog and content indexes are rebuilt by replaying JSONL through bulk
Practical Takeaways
Benchmark your own batch size. Elastic gives you a doubling procedure rather than a magic number precisely because the answer is workload-specific. Start at 100 documents and double until throughput stops improving.
Cap by bytes, not just by count. A batch of 10,000 small log lines and a batch of 10,000 large documents are very different requests. The documented ceiling is expressed in megabytes.
Always read the per-line response. Partial success is the design, not an edge case. Retry only the failed operations, and see JSONL best practices for error-handling patterns.
Apache Spark JSON Data Source
JSON Lines is the default; regular JSON is the special case
What the Documentation Says
The Spark SQL guide is unusually direct about this. Its JSON Files page warns: "Note that the file that is offered as a json file is not a typical JSON file. Each line must contain a separate, self-contained valid JSON object." The page then links out to the JSON Lines specification by name. In other words, when you call spark.read.json(), Spark is reading JSONL unless you tell it otherwise.
Why the Line Break Is Load-Bearing
A Split Needs a Record Boundary
Spark divides input files into splits and hands each split to a different executor. A newline is a boundary any worker can find without parsing what came before it, which is what makes a large JSONL file parallelizable across a cluster.
multiLine Gives That Up
Setting multiLine to true lets Spark read a conventional multi-line JSON document, but a single JSON value spanning a whole file has no interior boundary to split on. The documented option exists; the parallelism does not come with it.
Inference Costs a Pass
"Spark SQL can automatically infer the schema of a JSON dataset and load it as a DataFrame." Convenient, but inference has to read data to do its job. Supplying an explicit schema skips that work on every run.
Reading JSONL with Spark
PySpark and Scala
# Python / PySpark - JSONL is the default expectation
df = spark.read.json("s3://bucket/data/*.jsonl")
df.show()
# Scala
val df = spark.read.json("s3://bucket/data/*.jsonl")
df.show()
# Explicit schema: skips the inference pass
schema = StructType([
StructField("timestamp", StringType()),
StructField("user_id", IntegerType()),
StructField("event", StringType())
])
df = spark.read.schema(schema).json("data.jsonl")
# A conventional multi-line JSON document needs the opt-in
df = spark.read.option("multiLine", "true").json("nested.json")A Published Benchmark, With Its Conditions
NVIDIA published measured results for JSON processing in Spark using the RAPIDS Accelerator. These are real numbers from a named production workload on named hardware, which is exactly what makes them usable:
NVIDIA reports "the GPU runtime reduced from 16.7 hours to 3.8 hours, which is a 4x speedup and 80% cost savings" on a workload described as "large queries processing tens of terabytes of JSON data in a single Spark workload".
The cluster nodes are "GCP n1-standard-16 instances with a single NVIDIA T4 GPU attached to each node". Quote the number without the hardware and it stops meaning anything.
Source: NVIDIA Technical Blog - Accelerating JSON Processing on Apache Spark with GPUs
Where This Shows Up
- Data lakes: JSONL landing zones in object storage read directly by
spark.read.json() - ETL: JSONL in, Spark SQL transforms, columnar out - see analytics pipelines
- Training data prep: JSONL datasets loaded for distributed ML
- Structured Streaming: JSONL event streams consumed continuously
Practical Takeaways
Never hand Spark a single-line JSON array. The docs tell you plainly that this is not what the reader expects. One record per line is the contract.
Supply a schema once the shape is stable. Inference is a convenience for exploration, not a production setting.
Convert to a columnar format for repeated queries. JSONL is an excellent landing and interchange format. For the same analytical query run every day, Parquet or ORC is the better storage choice. See format comparisons.
ClickHouse JSONEachRow
A format whose documented aliases are JSONLines, NDJSON, and JSONL
What the Documentation Says
ClickHouse describes JSONEachRow as a format in which "ClickHouse outputs each row as a separated, newline-delimited JSON Object", and the format page lists JSONLines, NDJSON, and JSONL as accepted aliases for the same name. It works in both directions - as an input format for INSERT and as an output format for SELECT. ClickHouse's JSON loading guide goes further and calls NDJSON "the preferred format for loading JSON due to its brevity and efficient use of space".
Sources: ClickHouse - JSONEachRow format, ClickHouse - Loading JSON
The Trade-Off ClickHouse Publishes Openly
This is the most useful thing on this page, and it argues against JSONL. ClickHouse's insert-strategy guidance tells applications to prioritize performance-oriented formats and ranks them explicitly:
Native - recommended
"Most efficient. Column-oriented, minimal parsing required server-side." Used by default in the Go and Python clients.
RowBinary
"Efficient row-based format, ideal if columnar transformation is hard client-side." Used by the Java client.
JSONEachRow
"Easy to use but expensive to parse. Suitable for low-volume use cases or quick integrations." That is ClickHouse's own assessment, not a critic's.
The lesson is not that JSONL is bad. It is that JSONL buys you portability and legibility, and a binary columnar format buys you server-side parsing cost. A file-based import, an ad hoc backfill, or an integration you want any language to be able to produce is exactly where JSONL wins. A dedicated high-volume ingest client is where it does not.
Loading and Emitting JSONL
Documented SQL
-- Load a newline-delimited JSON file
INSERT INTO football FROM INFILE 'football.json' FORMAT JSONEachRow;
-- Read newline-delimited JSON straight from object storage
SELECT * FROM s3(
'https://datasets-documentation.s3.eu-west-3.amazonaws.com/pypi/json/*.json.gz',
JSONEachRow
);
-- Emit query results as JSONL
SELECT * FROM events FORMAT JSONEachRow;Related Documented Formats
- JSONCompactEachRow: "differs from JSONEachRow only in that data rows are output as arrays, not as objects" - smaller output when the column order is already known
- JSONEachRowWithProgress: output only. ClickHouse "will also yield progress information as JSON values", interleaving
progresslines among therowlines - Unknown fields are not skipped by default: columns with unknown names are only skipped when
input_format_skip_unknown_fieldsis set to 1
Documented Batching Guidance
"We recommend inserting data in batches of at least 1,000 rows, and ideally between 10,000-100,000 rows." Note how closely this mirrors Elastic's advice for a completely different engine.
"We recommend keeping the number of insert queries around one insert query per second." If the client cannot batch, asynchronous inserts move the buffering server-side.
Asynchronous Insert Defaults
With async_insert = 1, ClickHouse buffers inserts and flushes when any one of three documented thresholds is reached: a buffer size of async_insert_max_data_size (default 100 MiB), a time threshold of async_insert_busy_timeout_ms (default 200 ms, or 1000 ms on ClickHouse Cloud), or async_insert_max_query_number accumulated queries (default 450).
Practical Takeaways
Use JSONEachRow for imports and integrations. It is the documented preferred JSON format for loading, and it is readable by every language without a client library.
Switch to Native for sustained high-volume ingest. ClickHouse says so directly. Treating JSONL as universally optimal contradicts the vendor's own guidance.
Batch first, compress second. The docs pair batching with compression, recommending LZ4 for speed and ZSTD for compression ratio. More on this in the JSONL performance guide.
DuckDB read_ndjson
SQL over newline-delimited JSON, locally or in object storage
What the Documentation Says
DuckDB ships a dedicated reader for newline-delimited JSON. Its JSON loading page documents read_ndjson() as a JSON read with the format parameter set to newline_delimited, and defines that setting plainly: with it, "NDJSON is read, where each JSON is separated by a newline". The page links directly to the NDJSON specification. The filename argument "can also be a list of files, or a glob pattern", so a directory of daily JSONL files is a single query.
Querying JSONL Without a Pipeline
The practical appeal is that there is no ingestion step. You point SQL at the file and the file stays a file:
Documented Usage
-- Read a JSONL file directly
SELECT * FROM read_json('input.json');
-- Force newline-delimited parsing explicitly
SELECT * FROM read_json_objects('birds-nd.json', format = 'newline_delimited');
-- Materialize it, or append to an existing table
CREATE TABLE new_tbl AS SELECT * FROM read_json('input.json');
INSERT INTO tbl SELECT * FROM read_json('input.json');
COPY tbl FROM 'input.json';
-- Object storage, via the httpfs extension
INSTALL httpfs;
LOAD httpfs;
SELECT * FROM 's3://your-bucket/filename.extension';The S3 form above is DuckDB's own documented example. The httpfs docs state the mechanism "works for all files supported by DuckDB or its various extensions, and provides read-only access" over HTTP(S), with full read, write, and glob support against the S3 API.
Documented Reader Functions
- read_json(filename): "Read JSON from filename, where filename can also be a list of files, or a glob pattern"
- read_json_auto(filename): documented as an alias for
read_json, not a separate behaviour - read_ndjson / read_ndjson_auto: the same read with
formatpinned tonewline_delimited - read_ndjson_objects: returns each line as an unparsed JSON value, useful when the shape varies line to line
Published Measurements, With Their Caveats
DuckDB's engineering blog published concrete timings for JSONL work. These are laptop measurements from a March 2023 post, so treat them as an order-of-magnitude illustration rather than a throughput specification:
"If your JSON file is newline-delimited, DuckDB can parallelize reading", and "DuckDB will read multiple files in parallel". The parallelism is a property of the framing, not of JSON.
Counting 4.4 million GitHub Archive events. DuckDB states this "takes around 7.3 seconds on my laptop, a 2020 MacBook Pro with an M1 chip and 16 GB of memory", against 2.3 GB compressed and 18 GB uncompressed.
Source: DuckDB blog (March 2023) - Shredding Deeply Nested JSON, One Vector at a Time
Practical Takeaways
Newline-delimited input unlocks the parallel path. The same file content wrapped in a single JSON array does not get read the same way.
Glob instead of concatenating. Pointing the reader at a pattern beats a preprocessing step that merges files first.
This is the cheapest way to explore a JSONL dump. No server, no schema declaration, no load step. For hands-on practice see the JSONL examples.
Hugging Face Datasets
Line-per-row called out as the most efficient JSON layout
What the Documentation Says
The Datasets loading guide states the preference outright: "JSON files have diverse formats, but we think the most efficient format is to have multiple JSON objects; each line represents an individual row of data." Nested documents are still supported through the field argument, but the recommended shape is one record per line - JSONL by another name.
Sources: Hugging Face Datasets - Load, Hugging Face Datasets - Stream
The Problem Streaming Solves
Modern training corpora are larger than the machines that consume them, and the documentation gives a striking example of the gap:
The Dataset Does Not Fit
The docs note that "the English split of the HuggingFaceFW/fineweb dataset is 45 terabytes, but you can use it instantly with streaming". No disk on a normal workstation holds that, and downloading it first is not a plan.
Rows Are Not Flat
Training examples carry nested structures - message arrays, annotations, embeddings, provenance metadata. CSV flattens badly; one JSON object per line does not have to.
Iteration Needs a Boundary
To hand a training loop one example at a time over a network, the reader has to know where a record ends without having seen the rest of the file. That is exactly what a newline provides.
Loading and Streaming
Documented Behaviour
- One call to load:
load_dataset("json", data_files=...)handles local paths, HTTPS URLs, andhf://repository paths - Streaming is a flag: "Dataset streaming lets you work with a dataset without downloading it. The data is streamed as you iterate over the dataset."
- A different type, deliberately: streaming "creates a new dataset type instance (instead of the classic Dataset object), known as an IterableDataset"
- Compressed shards work directly: the streaming docs point the JSON loader at a glob of
.jsonl.gzfiles
Documented Calls
from datasets import load_dataset
# One object per line - the recommended layout
dataset = load_dataset("json", data_files="my_file.json")
# Nested document: name the field holding the rows
dataset = load_dataset("json", data_files="my_file.json", field="data")
# Stream compressed JSONL shards without downloading them
data_files = {'train': 'path/to/OSCAR-2201/compressed/en_meta/*.jsonl.gz'}
dataset = load_dataset('json', data_files=data_files, split='train', streaming=True)The recommended line-per-row layout, as printed in the docs:
{"a": 1, "b": 2.0, "c": "foo", "d": false}
{"a": 4, "b": -5.5, "c": null, "d": true}Practical Takeaways
Publish datasets line-per-row. It is what the loader is optimized for and what the documentation explicitly recommends.
Shard and compress. Many .jsonl.gz files beat one enormous file: they stream, they glob, and they parallelize.
Streaming changes the object you get back. An IterableDataset does not support random access, so code written against an index will need rewriting. More at JSONL for machine learning.
OpenAI File Pipelines
JSONL is mandatory for fine-tuning and for batch jobs
What the Documentation Says
Two separate OpenAI products require the same file format. The supervised fine-tuning guide instructs you to "use JSONL format, with one complete JSON structure on every line of the training data file". The Batch API guide describes a job as starting from "a .jsonl file where each line contains the details of an individual request to the API". Neither accepts a JSON array.
Sources: OpenAI - Supervised fine-tuning, OpenAI - Batch API
Documented Limits Worth Knowing
Example Counts
"The minimum number of examples you can provide for fine-tuning is 10." The same guide adds: "We see improvements from fine-tuning on 50-100 examples, but the right number for you varies greatly and depends on the use case."
Per-Example Context
The fine-tuning best-practices page publishes a per-model examples-context limit of 65,536 tokens for the current GPT-4.1 and GPT-4o families, and warns that "examples longer than the default are truncated to the maximum context length, which removes tokens from the end of the training example". A silently truncated line is still a valid JSONL line.
File Storage
The Files API reference states that "individual files can be up to 512 MB, and each project can store up to 2.5 TB of files in total". That is the general Files API limit, not a fine-tuning-specific one - the 200 MB figure above applies specifically to Batch input files.
Line Identity in Batches
Each batch request "must include a unique custom_id value, which you can use to reference results after completion", because output line order may not match input line order. The line number is not the identifier.
Sources: OpenAI - Fine-tuning best practices, OpenAI - Files API reference
The Two File Shapes
Fine-Tuning: One Conversation Per Line
{"messages": [{"role": "system", "content": "Marv is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What's the capital of France?"}, {"role": "assistant", "content": "Paris"}]}Structure as documented on OpenAI's fine-tuning best-practices page. Note the entire conversation is one line.
Batch: One API Request Per Line
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo-0125", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello world!"}], "max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo-0125", "messages": [{"role": "system", "content": "You are an unhelpful assistant."}, {"role": "user", "content": "Hello world!"}], "max_tokens": 1000}}Example structure from OpenAI's Batch API guide. The file is uploaded with its purpose set to batch.
Practical Takeaways
Validate every line before uploading. A file rejected on line 40,000 wastes the whole upload. Run it through the JSONL validator first.
Carry your own identifier. Batch results are correlated by custom_id, not by position. Any pipeline that assumes stable ordering will silently mismatch results.
Watch the byte ceiling as well as the line count. 50,000 requests and 200 MB are separate limits, and long prompts hit the second one first.
Amazon Data Firehose & BigQuery
One service adds the delimiter; the other refuses data without it
What the Documentation Says
These two services sit on opposite ends of the same pipeline, and between them they make the case for JSONL better than any benchmark could. AWS built a dedicated processor whose only job is to insert the newline. Google refuses to load JSON that lacks it. If you stream events into object storage and then load them into a warehouse, the format is effectively chosen for you.
Two Halves of One Pipeline
Amazon Data Firehose - Writing
- A processor exists purely for this: "If you want to add a new line delimiter between records in objects that are delivered to Amazon S3, choose AppendDelimiterToRecord as a processor type"
- Zero configuration: "You don't have to put a processor parameter when you select AppendDelimiterToRecord"
- It is a first-class type: the documented valid processor types are RecordDeAggregation, Decompression, CloudWatchLogProcessing, Lambda, MetadataExtraction, and AppendDelimiterToRecord
- Why it matters: without it, buffered records are concatenated into one S3 object with no boundary between them, and every downstream reader has to guess
Google BigQuery - Reading
- It is a requirement, not a preference: "JSON data must be newline-delimited, or ndJSON"
- One object per line: "Each JSON object must be on a separate line in the file"
- Top-level arrays are out: the documented form precludes wrapping your records in JSON array syntax for a load job
- Nested data still works: nested and repeated fields are supported inside each line, so flattening is not the price of admission
Illustrative Sensor Records (JSONL)
{"sensor_id":"temp_sensor_42","timestamp":"2026-01-15T14:32:15.123Z","temperature":72.5,"humidity":45.2,"location":{"building":"HQ","floor":3,"room":"3A"},"metadata":{"firmware":"v2.1.0","battery_pct":87}}
{"sensor_id":"motion_sensor_18","timestamp":"2026-01-15T14:32:16.456Z","motion_detected":true,"confidence":0.95,"location":{"building":"HQ","floor":2,"room":"2B"},"metadata":{"firmware":"v1.8.3","battery_pct":92}}
{"sensor_id":"temp_sensor_42","timestamp":"2026-01-15T14:32:45.789Z","temperature":72.7,"humidity":45.0,"location":{"building":"HQ","floor":3,"room":"3A"},"metadata":{"firmware":"v2.1.0","battery_pct":87}}Illustrative sample written for this page, not a captured production feed. It shows the nested-within-a-line shape both services accept.
Practical Takeaways
Turn the delimiter processor on at stream-creation time. Retrofitting it does not repair objects already written without boundaries, and repairing those files later is genuinely unpleasant.
Do not build a load job around a JSON array. BigQuery's wording is a hard requirement. Emit JSONL from the start rather than converting at the boundary. See JSONL for data streaming.
Partition on write. Dynamic partitioning groups delivered records under S3 prefixes by keys in the data, which is what makes the resulting JSONL cheap to query later.
What These Platforms Have in Common
Seven independent engineering teams, no shared roadmap, and the same four conclusions. These are patterns drawn from the documentation quoted above, not measured averages.
The Newline Is What Parallelizes
Spark requires "a separate, self-contained valid JSON object" per line so it can split a file across executors. DuckDB states plainly that parallel reading is conditional on the file being newline-delimited. In both cases the gain comes from the framing, not from JSON.
Batching Is the Documented Lever
Elastic says bulk requests "yield much better performance than single-document index requests" and tells you to find your size by doubling from 100. ClickHouse recommends 10,000 to 100,000 rows per insert. Two unrelated engines, the same advice: amortize the per-request cost, then measure.
You Can Consume It Without Downloading It
Hugging Face streams a 45 terabyte split "instantly" because each line is independently consumable. Line-delimited framing is what lets a reader begin work before the file ends - and what lets a producer append without rewriting.
It Has Become the Interchange Default
BigQuery requires ndJSON for load jobs. OpenAI requires JSONL for both fine-tuning and batch. AWS ships a processor whose only purpose is adding the delimiter. When independent vendors converge on a format, portability stops being a nice-to-have.
And Where JSONL Is the Wrong Answer
A page of case studies that only ever concludes "use JSONL" is marketing. The documentation quoted above contains two clear counterexamples, and they are worth more than another success story:
- Sustained high-volume ingest. ClickHouse calls JSONEachRow "easy to use but expensive to parse" and recommends its binary Native format for applications that care about insert performance. Text parsing is a real server-side cost.
- Repeated analytical queries over the same data. A row-oriented text format has to read every field of every line. For a dashboard query run hourly against a fixed dataset, a columnar format such as Parquet is the better storage choice - JSONL is the landing and interchange format, not necessarily the resting one.
The consistent pattern across all seven platforms is narrower and more durable than "JSONL is fast": JSONL is the format things are moved and shared in, because a newline is a boundary every language, every reader, and every vendor already agrees on.
Put JSONL to Work in Your Own Pipeline
Every platform above accepts the same file. Validate yours, then start moving data.