<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0"><channel><title><![CDATA[GeoJSONL — Interline Technologies]]></title><description><![CDATA[GeoJSONL is a readable and fast data format for geospatial vector data.]]></description><link>https://www.interline.io/</link><image><url>https://www.interline.io/favicon.ico</url><title>GeoJSONL — Interline Technologies</title><link>https://www.interline.io/</link></image><lastBuildDate>Fri, 08 Mar 2019 08:00:00 GMT</lastBuildDate><atom:link href="https://www.interline.io/blog/tag/geojsonl/rss.xml" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[Even more geospatial tools supporting the "GeoJSONL" format]]></title><description><![CDATA["GeoJSONL" has the easy readability of a GeoJSON file without the the performance limitations of the format. Last year we started producing GeoJSONL files from our OSM Extracts service. Now a growing list of long-existing and brand-new geospatial tools.]]></description><link>https://www.interline.io/blog/here-cli-support-geojsonl/</link><guid isPermaLink="false">6725410eb62841000136ec69</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 08 Mar 2019 08:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/download-geojsonl-extract.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/download-geojsonl-extract.png" alt="Even more geospatial tools supporting the &quot;GeoJSONL&quot; format"><p>Last year we expanded our <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> service to produce what we call GeoJSONL. This is a data format that goes by many names: newline-delimited GeoJSON, line-oriented GeoJSON, GeoJSONSeq, or GeoJSON Text Sequences. Whatever its name, the format enables effficient reading and efficient writing of <a href="https://en.wikipedia.org/wiki/GIS_file_formats#Vector">vector geo-data</a> features.</p><p>For more background, see our <a href="https://www.interline.io/blog/geojsonl-extracts/">previous blog post on GeoJSONL</a> and see this new <a href="https://stevage.github.io/ndgeojson/">overview of the format’s many names and its defining properties</a>.</p><p>Over the last year, there’s been even more progress on tools for writing and reading GeoJSONL (or whatever your preferred name is for this format).</p><h2 id="geojsonl-support-in-here-xyz">GeoJSONL support in HERE XYZ</h2><p>HERE XYZ now supports GeoJSONL ingest.</p><p>Don’t know what HERE XYZ is? Here’s how we summarized the platform in a <a href="https://www.interline.io/blog/here-xyz-open-geo-data/">blog post</a> last year:</p><blockquote><p>HERE XYZ provides the full set of components that &quot;modern&quot; geo developers have come to expect: GeoJSON input and output, &quot;slippy map&quot; libraries, a RESTful API, a JavaScript SDK, a cross-platform CLI, base maps and styles. As we've been experimenting with the HERE XYZ beta, we've found it provides some unique combinations that make it especially well suited to combining open geodata APIs — especially in large quantities.</p></blockquote><p>Using the latest version of the <a href="https://www.npmjs.com/package/@here/cli">HERE CLI</a>, you can parse and upload GeoJSONL files to the HERE XYZ platform.</p><p>Here’s an update of our <a href="https://interline-io.github.io/here-xyz-tutorials/tutorial-1/">“Use Crowdsourced Data from Interline OSM Extracts in an XYZ Web Map” tutorial</a> that uses the GeoJSONL format.</p><p>Download a GeoJSONL extract from OSM Extracts:</p><figure><img src="https://www.interline.io/content/images/2024/11/download-geojsonl-extract-1.png" alt="terminal showing usage of HERE CLI to upload OSM Extract in GeoJSONL format" width="2000" height="1085"></figure><p>Upload to XYZ and set tags:</p><figure><img src="https://www.interline.io/content/images/2024/11/here-cli-upload-geojsonl.png" alt="screenshot of a terminal window" width="1688" height="502"></figure><p>Previously you might hit memory limits on your computer if you tried to upload an extract for a big metro region. Now, thanks to GeoJSONL’s efficient parsing, you should be able to upload any extract of any size!</p><h2 id="the-growing-toolchain">The growing toolchain</h2><p>There’s a long history of tools that work with this format, and it’s been growing over the last year. Here are some more highlights:</p><p><strong>Command line</strong> On the command line, read <code>.geojsonl</code> and <code>.geojsons</code> files using GDAL’s <a href="https://gdal.org/drivers/vector/geojsonseq.html"><code>drv_geojsonseq</code></a>.</p><p><strong>Desktop GIS</strong> GDAL also powers <a href="https://www.qgis.org/">QGIS</a>, so now you can load <code>.geojsonl</code> and <code>.geojsons</code> files into our favorite desktop GIS sofware.</p><p><strong>Parse in Python</strong>: <a href="https://github.com/Toblerity/Fiona">Fiona</a> is a Python library that wraps GDAL and provides a “neat and nimble” interface. The next release will include <a href="https://github.com/Toblerity/Fiona/issues/720">support</a> for <code>drv_geojsonseq</code>.<a href="https://www.interline.io/blog/here-cli-support-geojsonl/#footnote-1">*</a></p><p><strong>Statistics and data science in R</strong> If you’re loading vector geo-data into R, try the <a href="https://cran.r-project.org/package=geojson">geojson</a> package on CRAN. See these <a href="https://github.com/ropensci/geojson#newline-delimited-geojson">instructions</a> for how to load a <code>.geojsonl</code> file directly from Interline OSM Extracts.</p><p><strong>Convert from a shapefile</strong> Install the <a href="https://www.npmjs.com/package/shapefile">shapefile NPM package</a> and run <code>shp2json --newline-delimited</code> to convert a shapefile to GeoJSONL. See these <a href="https://github.com/mbostock/shapefile#shp2json_newline_delimited">instructions</a>.</p><p><strong>Convert to vector tiles</strong> <a href="https://github.com/mapbox/tippecanoe/">Tippecanoe</a> can take GeoJSONL as input and turn it in to <a href="https://github.com/mapbox/awesome-vector-tiles">MVT vector tiles</a>. It also provides its own <code>tippecanoe-json-tool</code> command to convert a standard GeoJSON file to GeoJSONL format (see <a href="https://github.com/mapbox/tippecanoe/#tippecanoe-json-tool">these instructions</a>).</p><p><strong>Stream in to a web browser</strong> Watch a GeoJSONL file stream into a web in this <a href="https://observablehq.com/@bcamper/streaming-geojsonl">interactive JavaScript notebook</a>.<a href="https://www.interline.io/blog/here-cli-support-geojsonl/#footnote-2">†</a> If you have an Interline OSM Extracts API key and an Observable account, try <a href="https://observablehq.com/@drewda/streaming-geojsonl-from-your-choice-of-openstreetmap-extr">this notebook</a> to select from any of the 200+ cities and watch the stream arrive:</p><figure><img src="https://www.interline.io/content/images/2024/11/streaming-geojsonl.jpg" alt="" width="600" height="547"></figure><h2 id="when-to-try-geojsonl">When to try GeoJSONL</h2><p>If you like GeoJSON but find your data files are too big to hold in memory, give one of these tools a try.</p><p>If you need to run a bunch of vector geo-data through a <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> operation that, give this format a try. (It’s easy to divide a GeoJSONL file in to chunks before parsing its exact contents.)</p><p>If you want to download <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> from Interline and explore even the largest of extracts using HERE XYZ, give this a try.</p><hr><h3 id="notes">Notes</h3><p><a id="footnote-1"></a></p><p><sup>*</sup> Shoutout to <a href="https://github.com/sgillies">Sean Gillies</a>, the lead creator of Fiona and instigator of <a href="https://tools.ietf.org/html/rfc8142">RFC 8142 - GeoJSON Text Sequences</a>. We may have differing views on the <a href="https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text">record separator character</a>. Regardless, his packages like Fiona and <a href="https://rasterio.readthedocs.io/">Rasterio</a> are fabulous and we’re happy users of them.</p><p><a id="footnote-2"></a></p><p><sup>†</sup> Shoutout to <a href="http://vector.io/">Brett Camper</a>, the author of this Observable notebook and a fellow alum of <a href="https://mapzen.com/">Mapzen</a>.</p>]]></content:encoded></item><item><title><![CDATA[GeoJSONL: An optimized format for large geographic datasets]]></title><description><![CDATA[GeoJSON Lines (geojsonl) is a simple, newline-delimited variant of GeoJSON that allows large datasets to be loaded with a much lower memory footprint and easily integrate with traditional text-based tools. Interline OSM Extracts now provides geojsonl, along with a small but growing number of tools.]]></description><link>https://www.interline.io/blog/geojsonl-extracts/</link><guid isPermaLink="false">67253e7db62841000136ec47</guid><dc:creator><![CDATA[Ian Rees]]></dc:creator><pubDate>Tue, 11 Sep 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/preview.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/preview.png" alt="GeoJSONL: An optimized format for large geographic datasets"><blockquote><p><strong>March 8, 2019</strong>: We've published a follow-up blog post: <a href="https://www.interline.io/blog/here-cli-support-geojsonl/">Even more geospatial tools supporting the &quot;GeoJSONL&quot; format</a>. Now you can stream GeoJSONL files up to HERE XYZ, down to a web browser, or between your own programs using a growing list of software libraries.</p></blockquote><p><a href="http://geojson.org/">GeoJSON</a> is a popular format for exchanging geographic data. It builds on the success and flexibility of JSON to describe structured data, and simplifies many of the more complicated aspects of traditional GIS file formats. Initially developed through an ad hoc process, GeoJSON is now formalized by <a href="https://tools.ietf.org/html/rfc7946">IETF RFC 7946</a>, with a <a href="https://datatracker.ietf.org/wg/geojson/charter/">working group</a> to guide future additions. The universality of the underlying JSON format and the <a href="https://en.wikipedia.org/wiki/World_Geodetic_System">WGS84</a> coordinate system make GeoJSON nearly ideal for web-based mapping projects.</p><p>However, very large data sets are common in GIS, and the structure of a GeoJSON file generally requires the entire file to be read into memory and decoded all at once. A large file (e.g. 1 Gb) can easily exhaust the resources of a desktop computer. While streaming JSON parsers exist, they are generally not part of standard libraries and introduce yet another dependency and complexity.</p><p>One potential solution to the problem of large JSON files is to restructure the file as an array of objects, rather than having a single root object. Because the JSON specification prohibits the newline <code>\n</code> character from being used in string literals without escaping, it opens the possibility of having one JSON object per line in a way that is compatible with both standard UNIX text-processing tools and can be read line-by-line with existing JSON parsers.</p><p>This convention has arisen independently in many contexts, and is known as <a href="http://ndjson.org/">Newline Delimited JSON</a> (<code>ndjson</code>), <a href="http://jsonlines.org/">JSON Lines</a> (<code>jsonl</code>), or <a href="https://tools.ietf.org/html/rfc8142">GeoJSON Text Sequences</a>.<a href="#footnote-1">*</a> For GeoJSON specifically, the root level <code>FeatureCollection</code> object is replaced with a simple array of features, one per line. This file can then be read line-by-line, feature-by-feature, and easily integrated with other tools that use newline-delimited records such as <a href="https://www.gnu.org/software/parallel/">GNU parallel</a>. Newline-delimited GeoJSON (GeoJSONL) has been casually <a href="https://macwright.org/2015/03/23/geojson-second-bite.html#streaming">proposed</a> several times before, and is natively supported by some tools such as the <a href="https://docs.osmcode.org/osmium/latest/osmium-export.html">Osmium</a> <code>export</code> command, Mapbox’s <a href="https://github.com/mapbox/tippecanoe">Tippecanoe</a>, and <a href="https://stedolan.github.io/jq/">jq</a>, the Swiss-army knife of JSON processors.</p><p>At Interline, our <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extract service</a> now provides these files as well as traditional GeoJSON and OSM PBF formats, with the goal of providing a simple basis for reading and filtering OSM extracts without the need for more complicated and specialized libraries. Here is an <a href="https://storage.googleapis.com/osm-extracts.interline.io/honolulu_hawaii.geojsonl">example GeoJSONL</a> file, which you can compare to <a href="https://storage.googleapis.com/osm-extracts.interline.io/honolulu_hawaii.geojson">regular GeoJSON</a>.</p><p>While the file sizes are nearly identical, reading and parsing the entire ~27 Mb GeoJSON file requires several seconds and ~240 Mb of memory. Iterating through the GeoJSONL file line-by-line takes a similar amount of time, but uses negligible amounts of memory:</p><pre><code><span><span>import json
</span></span><span><span>with open('honolulu_hawaii.geojsonl') as f:
</span></span><span><span>    for feature in f:
</span></span><span><span>        print json.loads(feature)
</span></span></code></pre><p>The above Python snippet also demonstrates that no additional dependencies are required to read the GeoJSONL file; a line iterator and parsing each row separately works well. Still, <a href="http://ndjson.org/libraries.html">libraries are available</a> for many languages that provide native support.</p><p>As mentioned above, GeoJSONL also provides excellent input for traditional text-processing tools. The following (contrived) example runs <code>jq</code> to find the frequency of <code>name</code>s for features with the <code>highway</code> tag, without loading the entire file into memory:</p><pre><code><span><span># cat honolulu_hawaii.geojsonl | jq 'select(.properties.highway!=null) | .properties.name' | sort | uniq -c | sort -n
</span></span><span><span>..</span><span>.snip.</span><span>..
</span></span><span><span>  36</span><span> &quot;State Highway 83&quot;
</span></span><span><span>  49</span><span> &quot;John A. Burns Freeway&quot;
</span></span><span><span>  59</span><span> &quot;Makakilo Drive&quot;
</span></span><span><span> 106</span><span> &quot;Farrington Highway&quot;
</span></span><span><span> 107</span><span> &quot;Kamehameha Highway&quot;
</span></span></code></pre><p>Given the increasing complexity and importance of geographic data in nearly every domain, and with increasing file sizes, GeoJSONL is useful and timely. We’re glad to now support the format in <a href="https://app.interline.io/osm_extracts/interactive_view">Interline OSM Extracts</a> and we encourage other tools to offer native support when possible.</p><hr><p><a id="footnote-1"></a></p><h2 id="notes">Notes</h2><p><sup>*</sup> Minor differences exist between <code>ndjson</code> and <code>jsonl</code>.</p><p>GeoJSON Text Sequences adds a record separator (RS) character, in addition to the newline used by <code>ndjson</code> and <code>jsonl</code>.</p><p>Based on these existing options, we recommend the use of <code>ndjson</code>, which has a <a href="https://github.com/ndjson/ndjson-spec">simple spec document</a> and allows the presence of blank lines. This supports the widest range of tools for parsing and consuming. We use <code>geojsonl</code> as the file extension, although this is just a matter of taste.</p><style>html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}</style>]]></content:encoded></item></channel></rss>