https://jason-grey.com/tags/apache/Recent content in Apache on jason greyHugoenThu, 13 Mar 2025 00:00:00 +0000https://jason-grey.com/posts/2025/airflow-at-scale/Thu, 13 Mar 2025 00:00:00 +0000https://jason-grey.com/posts/2025/airflow-at-scale/<h1 id="scaling-airflow-dataset-scheduling-lessons-from-common-crawl"> Scaling Airflow Dataset Scheduling: Lessons from Common Crawl <a class="heading-link" href="#scaling-airflow-dataset-scheduling-lessons-from-common-crawl"> <i class="fa-solid fa-link" aria-hidden="true" title="Link to heading"></i> <span class="sr-only">Link to heading</span> </a> </h1> <p>Since joining <a href="https://www.commoncrawl.org/" class="external-link" target="_blank" rel="noopener">Common Crawl</a>, I&rsquo;ve been involved with a number of activities revolving around indexing and cataloging the integrity of our data. To coordinate these activites, and to be able to re-run them upon dataset changes, I decided to give Airflow a try. The introduction of dataset-based scheduling in Airflow 2.4 seemed attractive, but it also comes with interesting challenges when working at our scale.</p>