qsv

qsv: Blazing-fast CSV data-wrangling toolkit

	Table of Contents
<br/>Hi-ho "Quicksilver" away!<br/><sub><sup>logo details</sup></sub><br/>	qsv is a command line program for querying, slicing,<br>indexing, analyzing, filtering, enriching, transforming,<br>sorting, validating, joining & converting CSV files.<br>Commands are simple, composable & "blazing fast".<br><br>* Commands<br>* Installation Options<br> * Whirlwind Tour / Notebooks / Lessons & Exercises<br>* Cookbook<br>* FAQ<br>* Performance Tuning<br>* 👉 Benchmarks 🚀<br>* Environment Variables<br>* Feature Flags<br>* Goals/Non-goals<br>* Testing<br>* NYC School of Data 2022/csv,conf,v8 slides<br>* Sponsor

Table of Contents

Hi-ho "Quicksilver" away! logo details

qsv is a command line program for querying, slicing, indexing, analyzing, filtering, enriching, transforming, sorting, validating, joining & converting CSV files. Commands are simple, composable & "blazing fast". * Commands * Installation Options * Whirlwind Tour / Notebooks / Lessons & Exercises * Cookbook * FAQ * Performance Tuning * 👉 Benchmarks 🚀 * Environment Variables * Feature Flags * Goals/Non-goals * Testing * NYC School of Data 2022/csv,conf,v8 slides * Sponsor

</div> <div align="center">

Try it out at qsv.dathere.com!

</div>

<a name="available-commands">Command	Description
apply<br>✨🚀🧠🤖🔣👆	Apply series of string, date, math & currency transformations to given CSV column/s. It also has some basic NLP functions (similarity, sentiment analysis, profanity, eudex, language & name gender) detection.
<a name="applydp_deeplink"></a>applydp<br>🚀🔣👆	applydp is a slimmed-down version of `apply` with only Datapusher+ relevant subcommands/operations (`qsvdp` binary variant only).
behead	Drop headers from a CSV.
cat<br>🗄️	Concatenate CSV files by row or by column.
clipboard	Provide input from the clipboard or save output to the clipboard.
count<br>📇🏎️🐻‍❄️	Count the rows in a CSV file. (11.87 seconds for a 15gb, 27m row NYC 311 dataset without an index. Instantaneous with an index.) If the `polars` feature is enabled, uses Polars' multithreaded, mem-mapped CSV reader for fast counts even without an index
datefmt<br>🚀👆	Formats recognized date fields (19 formats recognized) to a specified date format using strftime date format specifiers.
dedup<br>🤯🚀👆	Remove duplicate rows (See also `extdedup`, `extsort`, `sort` & `sortcheck` commands).
describegpt<br>🌐🤖🪄	Infer extended metadata about a CSV using a GPT model from OpenAI's API or an LLM from another API compatible with the OpenAI API specification such as Ollama or Jan.
diff<br>🚀	Find the difference between two CSVs with ludicrous speed!<br/>e.g. compare two CSVs with 1M rows x 9 columns in under 600ms!
enum<br>👆	Add a new column enumerating rows by adding a column of incremental or uuid identifiers. Can also be used to copy a column or fill a new column with a constant value.
excel<br>🚀	Exports a specified Excel/ODS sheet to a CSV file.
exclude<br>📇👆	Removes a set of CSV data from another set based on the specified columns.
explode<br>🔣👆	Explode rows into multiple ones by splitting a column value based on the given separator.
extdedup<br>	Remove duplicate rows from an arbitrarily large CSV/text file using a memory-mapped, on-disk hash table. Unlike the `dedup` command, this command does not load the entire file into memory nor does it sort the deduped file.
extsort<br>🚀	Sort an arbitrarily large CSV/text file using a multithreaded external merge sort algorithm.
fetch<br>✨🧠🌐	Fetches data from web services for every row using HTTP Get. Comes with HTTP/2 adaptive flow control, jql JSON query language support, dynamic throttling (RateLimit) & caching with available persistent caching using Redis or a disk-cache.
fetchpost<br>✨🧠🌐	Similar to `fetch`, but uses HTTP Post. (HTTP GET vs POST methods)
fill<br>👆	Fill empty values.
fixlengths	Force a CSV to have same-length records by either padding or truncating them.
flatten	A flattened view of CSV records. Useful for viewing one record at a time.<br />e.g. `qsv slice -i 5 data.csv \| qsv flatten`.
fmt	Reformat a CSV with different delimiters, record terminators or quoting rules. (Supports ASCII delimited data.)
frequency<br>📇😣🏎️👆🪄	Build frequency tables of each column. Uses multithreading to go faster if an index is present.
geocode<br>✨🧠🌐🚀🔣👆	Geocodes a location against an updatable local copy of the Geonames cities database. With caching and multi-threading, it geocodes up to 360,000 records/sec!
headers<br>🗄️	Show the headers of a CSV. Or show the intersection of all headers between many CSV files.
index	Create an index (📇) for a CSV. This is very quick (even the 15gb, 28m row NYC 311 dataset takes all of 14 seconds to index) & provides constant time indexing/random access into the CSV. With an index, `count`, `sample` & `slice` work instantaneously; random access mode is enabled in `luau`; and multithreading (🏎️) is enabled for the `frequency`, `split`, `stats`, `schema` & `tojsonl` commands.
input	Read CSV data with special commenting, quoting, trimming, line-skipping & non-UTF8 encoding handling rules. Typically used to "normalize" a CSV for further processing with other qsv commands.
join<br>👆	Inner, outer, right, cross, anti & semi joins. Automatically creates a simple, in-memory hash index to make it fast.
joinp<br>✨🚀🐻‍❄️	Inner, outer, right, cross, anti, semi & asof joins using the Pola.rs engine. Unlike the `join` command, `joinp` can process files larger than RAM, is multithreaded, has join key validation, pre-join filtering, supports asof joins (which is particularly useful for time series data) & its output columns can be coalesced. However, `joinp` doesn't have an --ignore-case option.
json<br>👆	Convert JSON to CSV.
jsonl<br>🚀🔣	Convert newline-delimited JSON (JSONL/NDJSON) to CSV. See `tojsonl` command to convert CSV to JSONL.
<a name="luau_deeplink"></a><br>luau 👑<br>✨📇🌐🔣	Create multiple new computed columns, filter rows, compute aggregations and build complex data pipelines by executing a Luau 0.635 expression/script for every row of a CSV file (sequential mode), or using random access with an index (random access mode).<br>Can process a single Luau expression or full-fledged data-wrangling scripts using lookup tables with discrete BEGIN, MAIN and END sections.<br> It is not just another qsv command, it is qsv's Domain-specific Language (DSL) with numerous qsv-specific helper functions to build production data pipelines.
partition<br>👆	Partition a CSV based on a column value.
prompt	Open a file dialog to either pick a file as input or save output to a file.
pseudo<br>🔣👆	Pseudonymise the value of the given column by replacing them with an incremental identifier.
py<br>✨🔣	Create a new computed column or filter rows by evaluating a python expression on every row of a CSV file. Python's f-strings is particularly useful for extended formatting, [with the ability to evaluate Python expressions as