https://jason-grey.com/tags/commoncrawl/Recent content in Commoncrawl on jason greyHugoenMon, 01 Jan 2024 00:00:00 +0000https://jason-grey.com/posts/2024/commoncrawl/Mon, 01 Jan 2024 00:00:00 +0000https://jason-grey.com/posts/2024/commoncrawl/<p>I&rsquo;ve been doing public and private work on <a href="https://commoncrawl.org" class="external-link" target="_blank" rel="noopener">Common Crawl</a> — the open repository of web crawl data that underpins a huge amount of research and AI training.</p> <p>Two specific contributions:</p> <ul> <li><strong><a href="https://github.com/commoncrawl/cc-pyspark" class="external-link" target="_blank" rel="noopener">cc-pyspark</a></strong> — Added support for file-wise processing, enabling more efficient batch operations on the crawl corpus.</li> <li><strong><a href="https://github.com/commoncrawl/webarchive-indexing" class="external-link" target="_blank" rel="noopener">webarchive-indexing</a></strong> — Migrated legacy mrjob tasks to modern Spark jobs to process 9PB+ of crawl data.</li> </ul>