JSONL / JSON Lines Formatter

JSONL (JSON Lines) is a text format where each line is a valid JSON value (usually an object). Also known as NDJSON (Newline Delimited JSON) or JSON Lines. Rules: each line is a complete, valid JSON object. Lines are separated by newlines (\n). No trailing comma between lines. The file may have a trailing newline. Advantages over JSON arrays: streaming-friendly (read one line at a time without loading the full file), easy to append, easy to grep, memory-efficient for large datasets. Used by: Elasticsearch bulk API, OpenAI fine-tuning datasets, log aggregation, Clickhouse, BigQuery.

JSON array: [{...}, {...}, {...}] — must read the entire file before parsing, good for small datasets. JSONL: one object per line, stream-parseable line by line. Conversion is straightforward: JSONL → JSON array: wrap lines in [] and join with commas. JSON array → JSONL: stringify each element on its own line. Practical difference: a 10 GB JSON array requires 10 GB of RAM to parse; a 10 GB JSONL file requires only enough RAM to hold one record at a time. Tools: jq, jsonlines Python library, Clickhouse, Apache Kafka all support JSONL natively.

Line by line: const records = text.split("\n").filter(line => line.trim()).map(line => JSON.parse(line)). Stream processing: use fs.createReadStream() with readline interface in Node.js. readline.createInterface({ input: fileStream }).on("line", (line) => { if (line.trim()) process(JSON.parse(line)); }). For large files, use the streaming approach — never load the entire file into memory. Browser: fetch the file and use a TransformStream with line splitting. Error handling: wrap each JSON.parse() in try-catch to handle malformed lines.

OpenAI fine-tuning uses JSONL with a specific schema. Chat format: each line is {"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}. Completion format (legacy): each line is {"prompt": "...", "completion": " ..."}. Requirements: at least 10 examples, recommended 50-100+. File size: max 1 GB. Validation: use the openai tools.fine_tunes.prepare_data command. The "\n" token after the completion is important for the model to learn where responses end.

Command line: count lines: wc -l file.jsonl. Get first 10: head -n 10 file.jsonl. Get last 10: tail -n 10 file.jsonl. Filter with jq: cat file.jsonl | jq -c 'select(.status == "active")'. Python streaming: with open("file.jsonl") as f: for line in f: record = json.loads(line). Split by line count: split -l 1000 file.jsonl chunk_. Merge: cat chunk_* > merged.jsonl. Sort: sort -t'"status"' file.jsonl (sort by raw line, be careful). DuckDB: SELECT * FROM read_json_auto('file.jsonl', format='newline_delimited').