<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Huge Files, No Waiting]]></title><description><![CDATA[Notes from building UwView and UwDiff — opening, searching and comparing multi-GB logs, CSV and text files. Every benchmark comes with the machine, the data and]]></description><link>https://hugefiles.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Mon, 16 Nov 2026 18:28:45 GMT</lastBuildDate><atom:link href="https://hugefiles.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Comparing a CSV Too Big for Excel Against Last Month's Version — Key Matching on 10 GB With Excel, qsv diff and DuckDB]]></title><description><![CDATA[Excel stops at row 1,048,576. I timed key matching of two 10 GB CSVs (100M rows) with qsv diff, DuckDB and UwDiff on one Mac. Losses included.]]></description><link>https://hugefiles.hashnode.dev/comparing-a-csv-too-big-for-excel-against-last-month-s-version-key-matching-on-10-gb-with-excel-qsv-diff-and-duckdb</link><guid isPermaLink="true">https://hugefiles.hashnode.dev/comparing-a-csv-too-big-for-excel-against-last-month-s-version-key-matching-on-10-gb-with-excel-qsv-diff-and-duckdb</guid><category><![CDATA[csv]]></category><category><![CDATA[duckDB]]></category><category><![CDATA[data-engineering]]></category><category><![CDATA[command line]]></category><category><![CDATA[Benchmark]]></category><category><![CDATA[excel]]></category><dc:creator><![CDATA[Yuji Maruyama]]></dc:creator><pubDate>Fri, 13 Nov 2026 10:09:57 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af6d664e4ebcb8a15ddbe24/936d5328-4a44-4e42-90da-a33ef8b151d2.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>The sheet ends at row 1,048,576.</strong></p>
<p>Last month's ledger and this month's. You want to see what changed, so you open both in Excel — and get "This data set is too large for the Excel grid." As the warning says, Excel does not load anything past row 1,048,576. And even the part it does load is not always what was in the file.</p>
<p>So how do you match a CSV that does not fit in Excel against last month's version by a key column? I timed three tools from 1 GB up to 10 GB (about 100 million rows). (I develop UwDiff, one of the three, so read the numbers with that in mind — the losses are included.)</p>
<hr />
<h2>The short answer: Excel cuts the table before you can compare it. Key matching on 10 GB took 22 s with UwDiff, 33 s with qsv diff and 125–220 s with DuckDB</h2>
<ul>
<li><strong>Excel stops at row 1,048,576.</strong> Even with a 1.05-million-row (100 MB) CSV, the last few thousand rows were not loaded, and values such as <code>1-10</code> were turned into dates just by opening the file.</li>
<li><strong>To match by key without Excel</strong>, I measured qsv (<code>qsv diff</code>) and DuckDB (a join in SQL) on the same Mac. With two 10 GB files whose rows were in the same order, qsv diff took 32.56 s and DuckDB 124.66 s. With the rows in a different order, DuckDB took 220.06 s and qsv diff did not finish within 10 minutes.</li>
<li><strong>UwDiff</strong> matches by key regardless of row order with a single line: <code>uvd old.csv new.csv -key ID</code>. On 10 GB it took 21.52 s (same order) and 22.61 s (different order). Every tool's output matched the expected diff exactly.</li>
<li><strong>Where it loses:</strong> with rows in the same order, qsv diff is as fast at 1 GB, and it used less memory at every size.</li>
</ul>
<hr />
<h2>Three reasons Excel cannot do this comparison</h2>
<p>I opened a 1.05-million-row (100 MB) CSV in Excel (Microsoft 365) on the same MacBook Air (2026-10-23).</p>
<ol>
<li><strong>It stops at 1,048,576 rows.</strong> Excel warns that the data set is too large for the grid and that unloaded data will be lost if you save, and leaves the rest out. A VLOOKUP or XLOOKUP built on half a table cannot give the right answer.</li>
<li><strong>Opening a file changes values.</strong> <code>1-10</code> became a date, and <code>2025/02/21</code> became <code>2025/2/21</code>. Excel also asks each time whether to strip leading zeros or convert to scientific notation; forget to say no, and the values change before you compare anything.</li>
<li><strong>It does not detect the encoding.</strong> A UTF-8 CSV without a BOM was read as Shift-JIS, and all the Japanese text was garbled.</li>
</ol>
<p>You can work around 1 and 3 (load into the data model with Power Query, or import with an explicit encoding). But producing "which cell in which row changed" between two files is not what Excel is built for.</p>
<h2>The usual tools: qsv diff and DuckDB</h2>
<p>Several tools can match two tables by a key column without Excel. I picked two that run on the same Mac.</p>
<ul>
<li><strong>qsv</strong> (<code>qsv diff --key 0 old.csv new.csv</code>): a collection of CSV commands. <code>diff</code> matches two files by key and reports added, deleted and changed rows.</li>
<li><strong>DuckDB</strong>: a database that reads CSV files directly with SQL. You write the SQL that joins old and new on the key and picks out rows whose values differ.</li>
</ul>
<p>Both handle rows in a different order.</p>
<h2>With UwDiff: name one key column</h2>
<h3>In the window</h3>
<ol>
<li>Open UwDiff and put last month's CSV in "旧" (old) and this month's in "新" (new) — drag and drop works too. (The window is in Japanese; the <a href="https://uvp.y42u.net/en/uwdiff-manual-en/">manual</a> has the English names.)</li>
<li>Enter a column name such as <code>ID</code> as the <strong>key column</strong>. Rows are matched by key even if their order differs.</li>
<li>Click "比べる" (Compare).</li>
</ol>
<p>Only the changed cells are listed, as pairs of old and new values. Values are compared as strings, so leading zeros, long numbers and date-like values stay as they are.</p>
<h3>On the command line</h3>
<pre><code class="language-plaintext">uvd ledger_sep.csv ledger_oct.csv -key ID -o changes.csv
</code></pre>
<p>The output is a CSV of <code>kind, key, column, old value, new value</code>, with one row per added, deleted or changed cell. (The header and kind labels are in Japanese: <code>区分,ID,列,旧値,新値</code> and 追加 / 削除 / 変更.)</p>
<ul>
<li>The key can be several columns (<code>-key ID,Type</code>).</li>
<li>Columns are matched by header name, so the comparison still works if the column order changed between months.</li>
<li>Old and new can be in different encodings (for example Shift-JIS and UTF-8) — UwDiff detects and aligns them while reading.</li>
<li>Excel files that do fit in Excel can be matched the same way (<code>uvd ledger_sep.xlsx ledger_oct.xlsx -key ID</code>).</li>
</ul>
<p>Keep only the rows you want in the diff and pass it to <code>uvd apply</code> to apply just those changes to last month's CSV. Its encoding, delimiter and row order are preserved (<a href="https://uvp.y42u.net/en/uwdiff-manual-en/">manual, 3-3</a>).</p>
<h3>Measured at 1 GB, 5 GB and 10 GB</h3>
<p>All three tools ran on the same machine with the same test data. Every result below was checked against the expected diff and <strong>matched exactly</strong>.</p>
<ul>
<li>Machine: MacBook Air (Apple M4, 10 cores, 32 GB memory), macOS 26.3.1, data on an external SSD (USB)</li>
<li>Method: cache purged before every run (<code>sudo purge</code>), one run each. Seconds are <code>/usr/bin/time</code> real time. Anything over 10 minutes was stopped (2026-10-26)</li>
<li>Versions: UwDiff 1.0.0 (uvd), qsv 26.0.0, DuckDB 2.0.0</li>
<li>Test data: columns "point name, X, Y, Z, type, station number, observation date, note". NEW was made from OLD with deletions, additions, cell edits, renamed keys and so on (the expected diff for 10 GB is about 2.56 million rows). Encoding: UTF-8</li>
</ul>
<table>
<thead>
<tr>
<th>Size × 2 files</th>
<th>Row order</th>
<th align="right">UwDiff (s)</th>
<th align="right">qsv diff (s)</th>
<th align="right">DuckDB (s)</th>
</tr>
</thead>
<tbody><tr>
<td>1 GB (about 10.5 million rows)</td>
<td>same</td>
<td align="right"><strong>2.28</strong></td>
<td align="right">2.73</td>
<td align="right">4.05</td>
</tr>
<tr>
<td>1 GB</td>
<td>different</td>
<td align="right"><strong>2.27</strong></td>
<td align="right">5.87</td>
<td align="right">4.41</td>
</tr>
<tr>
<td>5 GB (about 52.7 million rows)</td>
<td>same</td>
<td align="right"><strong>10.88</strong></td>
<td align="right">17.15</td>
<td align="right">23.33</td>
</tr>
<tr>
<td>5 GB</td>
<td>different</td>
<td align="right"><strong>11.11</strong></td>
<td align="right">36.01</td>
<td align="right">24.39</td>
</tr>
<tr>
<td>10 GB (about 100 million rows)</td>
<td>same</td>
<td align="right"><strong>21.52</strong></td>
<td align="right">32.56</td>
<td align="right">124.66</td>
</tr>
<tr>
<td>10 GB</td>
<td>different</td>
<td align="right"><strong>22.61</strong></td>
<td align="right">stopped after 10 min</td>
<td align="right">220.06</td>
</tr>
</tbody></table>
<ul>
<li>UwDiff's time grows in proportion to size: 10 times the size took about 10 times as long (1 GB 2.28 s → 10 GB 21.52 s).</li>
<li>At 10 GB with rows in the same order, qsv diff took 1.5 times as long as UwDiff and DuckDB 5.8 times. With rows in a different order, DuckDB took 9.7 times as long.</li>
<li>With Shift-JIS (CP932) on both sides, UwDiff's 10 GB times were 20.35 s (same order) and 21.90 s (different order).</li>
</ul>
]]></content:encoded></item></channel></rss>