https://jason-grey.com/tags/tooling/Recent content in Tooling on jason greyHugoenFri, 15 Nov 2024 00:00:00 +0000https://jason-grey.com/posts/2024/gzinspector/Fri, 15 Nov 2024 00:00:00 +0000https://jason-grey.com/posts/2024/gzinspector/<p>I published an open source tool &ldquo;<a href="https://github.com/jt55401/gzinspector" class="external-link" target="_blank" rel="noopener">gzinspector</a>&rdquo; to inspect gzip streams - specifically those encoded with many chunks.</p> <blockquote> <p>A robust command-line tool for inspecting and analyzing GZIP/ZLIB compressed files. GZInspector provides detailed information about compression chunks, headers, and content previews with support for both human-readable and JSON output formats.</p> </blockquote> <p>I did this due to the work I&rsquo;ve been doing for <a href="https://commoncrawl.org/" class="external-link" target="_blank" rel="noopener">CommonCrawl</a> - specifically around processing &ldquo;<a href="https://github.com/webrecorder/pywb/wiki/CDX-Index-Format#zipnum-sharded-cdx" class="external-link" target="_blank" rel="noopener">ZipNum</a>&rdquo; format <a href="https://github.com/webrecorder/cdxj-indexer" class="external-link" target="_blank" rel="noopener">CDXJ</a> indexes.</p> <p>If you find it useful, let me know (reach out or star it on github.)</p>https://jason-grey.com/posts/2024/common-crawl-checker/Tue, 06 Feb 2024 00:00:00 +0000https://jason-grey.com/posts/2024/common-crawl-checker/<h1 id="enter-a-hostname-see-if-common-crawl-has-it"> Enter a hostname, see if common crawl has it <a class="heading-link" href="#enter-a-hostname-see-if-common-crawl-has-it"> <i class="fa-solid fa-link" aria-hidden="true" title="Link to heading"></i> <span class="sr-only">Link to heading</span> </a> </h1> <p>This checks <a href="https://www.commoncrawl.org/blog/november-december-2023-crawl-archive-now-available" class="external-link" target="_blank" rel="noopener">CC-MAIN-2023-50</a> - which was from November/December 2023. I may update this in future to check the latest, but, for now, that&rsquo;s what we have.</p> <p>Give it a try here:</p> <script> function checkURL() { document.getElementById('result').textContent = '...'; var urlToCheck = document.getElementById('urlInput').value; var apiUrl = 'https://api.jason-grey.com/check_url?url=' + encodeURIComponent(urlToCheck); fetch(apiUrl) .then(response => response.json()) .then(data => { document.getElementById('result').textContent = data.result; }) .catch(error => { console.error('Error:', error); document.getElementById('result').textContent = 'Error calling the service'; }); } </script> <input type="text" id="urlInput" placeholder="Enter domain name to check"> <button onclick="checkURL()">Check</button> <p id="result"></p>