<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0"><channel><title><![CDATA[OpenStreetMap — Interline Technologies]]></title><description><![CDATA[OpenStreetMap is the world's largest collaboratively created geospatial dataset of roads, paths, points of interest, and other geographic features. We contribute to OSM and use it to power many of our services.]]></description><link>https://www.interline.io/</link><image><url>https://www.interline.io/favicon.ico</url><title>OpenStreetMap — Interline Technologies</title><link>https://www.interline.io/</link></image><lastBuildDate>Fri, 08 Mar 2019 08:00:00 GMT</lastBuildDate><atom:link href="https://www.interline.io/blog/tag/openstreetmap/rss.xml" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[Even more geospatial tools supporting the "GeoJSONL" format]]></title><description><![CDATA["GeoJSONL" has the easy readability of a GeoJSON file without the the performance limitations of the format. Last year we started producing GeoJSONL files from our OSM Extracts service. Now a growing list of long-existing and brand-new geospatial tools.]]></description><link>https://www.interline.io/blog/here-cli-support-geojsonl/</link><guid isPermaLink="false">6725410eb62841000136ec69</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 08 Mar 2019 08:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/download-geojsonl-extract.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/download-geojsonl-extract.png" alt="Even more geospatial tools supporting the &quot;GeoJSONL&quot; format"><p>Last year we expanded our <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> service to produce what we call GeoJSONL. This is a data format that goes by many names: newline-delimited GeoJSON, line-oriented GeoJSON, GeoJSONSeq, or GeoJSON Text Sequences. Whatever its name, the format enables effficient reading and efficient writing of <a href="https://en.wikipedia.org/wiki/GIS_file_formats#Vector">vector geo-data</a> features.</p><p>For more background, see our <a href="https://www.interline.io/blog/geojsonl-extracts/">previous blog post on GeoJSONL</a> and see this new <a href="https://stevage.github.io/ndgeojson/">overview of the format’s many names and its defining properties</a>.</p><p>Over the last year, there’s been even more progress on tools for writing and reading GeoJSONL (or whatever your preferred name is for this format).</p><h2 id="geojsonl-support-in-here-xyz">GeoJSONL support in HERE XYZ</h2><p>HERE XYZ now supports GeoJSONL ingest.</p><p>Don’t know what HERE XYZ is? Here’s how we summarized the platform in a <a href="https://www.interline.io/blog/here-xyz-open-geo-data/">blog post</a> last year:</p><blockquote><p>HERE XYZ provides the full set of components that &quot;modern&quot; geo developers have come to expect: GeoJSON input and output, &quot;slippy map&quot; libraries, a RESTful API, a JavaScript SDK, a cross-platform CLI, base maps and styles. As we've been experimenting with the HERE XYZ beta, we've found it provides some unique combinations that make it especially well suited to combining open geodata APIs — especially in large quantities.</p></blockquote><p>Using the latest version of the <a href="https://www.npmjs.com/package/@here/cli">HERE CLI</a>, you can parse and upload GeoJSONL files to the HERE XYZ platform.</p><p>Here’s an update of our <a href="https://interline-io.github.io/here-xyz-tutorials/tutorial-1/">“Use Crowdsourced Data from Interline OSM Extracts in an XYZ Web Map” tutorial</a> that uses the GeoJSONL format.</p><p>Download a GeoJSONL extract from OSM Extracts:</p><figure><img src="https://www.interline.io/content/images/2024/11/download-geojsonl-extract-1.png" alt="terminal showing usage of HERE CLI to upload OSM Extract in GeoJSONL format" width="2000" height="1085"></figure><p>Upload to XYZ and set tags:</p><figure><img src="https://www.interline.io/content/images/2024/11/here-cli-upload-geojsonl.png" alt="screenshot of a terminal window" width="1688" height="502"></figure><p>Previously you might hit memory limits on your computer if you tried to upload an extract for a big metro region. Now, thanks to GeoJSONL’s efficient parsing, you should be able to upload any extract of any size!</p><h2 id="the-growing-toolchain">The growing toolchain</h2><p>There’s a long history of tools that work with this format, and it’s been growing over the last year. Here are some more highlights:</p><p><strong>Command line</strong> On the command line, read <code>.geojsonl</code> and <code>.geojsons</code> files using GDAL’s <a href="https://gdal.org/drivers/vector/geojsonseq.html"><code>drv_geojsonseq</code></a>.</p><p><strong>Desktop GIS</strong> GDAL also powers <a href="https://www.qgis.org/">QGIS</a>, so now you can load <code>.geojsonl</code> and <code>.geojsons</code> files into our favorite desktop GIS sofware.</p><p><strong>Parse in Python</strong>: <a href="https://github.com/Toblerity/Fiona">Fiona</a> is a Python library that wraps GDAL and provides a “neat and nimble” interface. The next release will include <a href="https://github.com/Toblerity/Fiona/issues/720">support</a> for <code>drv_geojsonseq</code>.<a href="https://www.interline.io/blog/here-cli-support-geojsonl/#footnote-1">*</a></p><p><strong>Statistics and data science in R</strong> If you’re loading vector geo-data into R, try the <a href="https://cran.r-project.org/package=geojson">geojson</a> package on CRAN. See these <a href="https://github.com/ropensci/geojson#newline-delimited-geojson">instructions</a> for how to load a <code>.geojsonl</code> file directly from Interline OSM Extracts.</p><p><strong>Convert from a shapefile</strong> Install the <a href="https://www.npmjs.com/package/shapefile">shapefile NPM package</a> and run <code>shp2json --newline-delimited</code> to convert a shapefile to GeoJSONL. See these <a href="https://github.com/mbostock/shapefile#shp2json_newline_delimited">instructions</a>.</p><p><strong>Convert to vector tiles</strong> <a href="https://github.com/mapbox/tippecanoe/">Tippecanoe</a> can take GeoJSONL as input and turn it in to <a href="https://github.com/mapbox/awesome-vector-tiles">MVT vector tiles</a>. It also provides its own <code>tippecanoe-json-tool</code> command to convert a standard GeoJSON file to GeoJSONL format (see <a href="https://github.com/mapbox/tippecanoe/#tippecanoe-json-tool">these instructions</a>).</p><p><strong>Stream in to a web browser</strong> Watch a GeoJSONL file stream into a web in this <a href="https://observablehq.com/@bcamper/streaming-geojsonl">interactive JavaScript notebook</a>.<a href="https://www.interline.io/blog/here-cli-support-geojsonl/#footnote-2">†</a> If you have an Interline OSM Extracts API key and an Observable account, try <a href="https://observablehq.com/@drewda/streaming-geojsonl-from-your-choice-of-openstreetmap-extr">this notebook</a> to select from any of the 200+ cities and watch the stream arrive:</p><figure><img src="https://www.interline.io/content/images/2024/11/streaming-geojsonl.jpg" alt="" width="600" height="547"></figure><h2 id="when-to-try-geojsonl">When to try GeoJSONL</h2><p>If you like GeoJSON but find your data files are too big to hold in memory, give one of these tools a try.</p><p>If you need to run a bunch of vector geo-data through a <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> operation that, give this format a try. (It’s easy to divide a GeoJSONL file in to chunks before parsing its exact contents.)</p><p>If you want to download <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> from Interline and explore even the largest of extracts using HERE XYZ, give this a try.</p><hr><h3 id="notes">Notes</h3><p><a id="footnote-1"></a></p><p><sup>*</sup> Shoutout to <a href="https://github.com/sgillies">Sean Gillies</a>, the lead creator of Fiona and instigator of <a href="https://tools.ietf.org/html/rfc8142">RFC 8142 - GeoJSON Text Sequences</a>. We may have differing views on the <a href="https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text">record separator character</a>. Regardless, his packages like Fiona and <a href="https://rasterio.readthedocs.io/">Rasterio</a> are fabulous and we’re happy users of them.</p><p><a id="footnote-2"></a></p><p><sup>†</sup> Shoutout to <a href="http://vector.io/">Brett Camper</a>, the author of this Observable notebook and a fellow alum of <a href="https://mapzen.com/">Mapzen</a>.</p>]]></content:encoded></item><item><title><![CDATA[Bringing Back the Spirit of Map Mashups: Combine Open Geodata APIs using HERE XYZ]]></title><description><![CDATA[Interline and HERE are collaborating to make Interline's open data services like OSM Extracts and the Transitland API easy to combine, mix, filter, analyze, and visualize using the new HERE XYZ platform. ]]></description><link>https://www.interline.io/blog/here-xyz-open-geo-data/</link><guid isPermaLink="false">6725453eb62841000136ecbc</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Mon, 01 Oct 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/xyz-tutorial-1-animation.jpg" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/xyz-tutorial-1-animation.jpg" alt="Bringing Back the Spirit of Map Mashups: Combine Open Geodata APIs using HERE XYZ"><p>At Interline, we spend much of our time with open geodata APIs. We create our own open geodata APIs (like <a href="https://www.transit.land/">Transitland</a>) and we use others (for example, whenever we need to throw a bunch of GeoJSON on a base map). At their best, open geodata APIs are fun — you get to combine pieces together to create something new. Then again, public APIs can have many limitations — combine together a few and you can end up with the worst of all worlds: duplicate data, misregistered overlays, a slow user experience, incompatible data formats… With all these mixed experiences in our minds, we’ve been playing with the initial beta of <a href="http://explore.xyz.here.com/">HERE XYZ</a>. It’s a brand new developer service from one of the oldest names in geodata.</p><p>HERE XYZ provides the full set of components that “modern” geo developers have come to expect: GeoJSON input and output, “slippy map” libraries, a RESTful API, a JavaScript SDK, a cross-platform CLI, base maps and styles. As we’ve been experimenting with the HERE XYZ beta, we’ve found it provides some unique combinations that make it especially well suited to combining open geodata APIs — especially in large quantities.</p><p>Combining geodata is nothing new. <a href="http://ladprofile.weebly.com/ian-mcharg-1920-2001.html">Ian McHarg</a>, a prominent environmental designer, printed maps on transparent plastic sheets and overlaid them atop each other, inspiring the idea of layers in GIS (<a href="https://en.wikipedia.org/wiki/Geographic_information_system">geographic information systems</a>):</p><figure><img src="https://www.interline.io/content/images/2024/11/ian-mcharg.jpg" alt="photo of Ian McHarg" width="1100" height="742"></figure><p><em>Image credit: University of Pennsylvania</em></p><p>More recently, Google Maps brought us <em>mashups</em>. At first, it involved hacking the Google Maps JavaScript code in order to overlay <a href="https://www.nytimes.com/2005/12/11/magazine/doityourself-cartography.html">crimes in Chicago</a> and <a href="http://www.housingmaps.com/">Craigslist house listings</a>:</p><figure><img src="https://www.interline.io/content/images/2024/11/housingmaps2.png" alt="screenshot of old housingmaps.com" width="1088" height="684"></figure><p><em>Image credit: housingmaps.com</em></p><p>Neither hacking nor reverse-engineering are required any more to combine multiple sources of geodata in a web browser. Companies like Google, Esri, Mapbox, Carto — also a now defunct start-up where we previously worked, called Mapzen — all offer accessible APIs for creating web maps. We’ve also been freed of many of the constraints on dataset size in 2005 mashups. Original mashups involved drawing points and other geographic features on top of base-map images, with a low limit on the number of these vector features that could be drawn at once. More recent techniques send both the base map and overlays as vector tiles. Instead of an image, a vector tile is effectively a square of data, which can be pre-generated by a server. Many vector tiles are then loaded and drawn by your web browser simultaneously.</p><p>We’ve left behind the constraints of mashups — and it also feels like we’ve forgotten the fun of mashups. Vector tiles help greatly with performance, but often add complexity to the overall set of steps required to combine together existing and new data sources on a web map or another type of visualization.</p><p>Our experiments with the HERE CLI and the XYZ API are bringing back that creative spirit of 2005 mashups. The XYZ platform generates vector tiles near instantly for any new imported data sources. The HERE CLI and XYZ API provide a range of ways to import GeoJSON files, connect to APIs, or stream in from other CLI tools. It’s the best of 2005 and 2018. If you use the SDKs to style your web maps to look like the transparency overlays in Ian McHarg’s book <em>Design With Nature</em>, it’s the best of 1969 proto-GIS, too.</p><p>We’re collaborating with the HERE XYZ team to turn our experiments into tutorials that demonstrate how Interline’s open geodata services can be combined, mixed, filtered, analyzed, and visualized. In the first tutorial, you’ll fetch and filter data from <a href="https://app.interline.io/osm_extracts/interactive_view">Interline OSM Extracts</a>:</p><figure><img src="https://www.interline.io/content/images/2024/11/xyz-tutorial-1-animation-2.jpg" alt="" width="918" height="756"></figure><p>In the second tutorial, you’ll query the <a href="https://www.transit.land/">Transitland</a> API repeatedly to build up an analysis and a web map of just not subway locations but also the frequency of how often each subway line operates. (Especially when working with geodata for transportation applications, both space and time matter.) We’ll show you how to do this for the entire planet:</p><figure><img src="https://www.interline.io/content/images/2024/11/xyz-tutorial-2-animation-3.jpg" alt="" width="800" height="526"></figure><p>The sample projects and tutorials take intermediate and advanced users on a tour through some of the unique features of XYZ and the HERE CLI:</p><ul><li>filtering Interline OSM Extracts using OpenStreetMap tags</li><li>writing scripts that stream geodata from external APIs into an XYZ space using the HERE CLI</li><li>using multiple API queries to aggregate together properties about the same features (identified by unique IDs, such as <a href="https://www.transit.land/documentation/concepts/onestop-id-scheme/">Transitland Onestop IDs</a>)</li><li>quickly building global tile sets</li></ul><p>We’re experimenting with additional features, for creating a new generation of mashups, and we look forward to sharing more soon.</p><p>Now, get started with our <a href="https://www.interline.io/docs/here-xyz/">first two tutorials on combining HERE XYZ, Interline OSM Extracts, and Transitland</a>.</p>]]></content:encoded></item><item><title><![CDATA[GeoJSONL: An optimized format for large geographic datasets]]></title><description><![CDATA[GeoJSON Lines (geojsonl) is a simple, newline-delimited variant of GeoJSON that allows large datasets to be loaded with a much lower memory footprint and easily integrate with traditional text-based tools. Interline OSM Extracts now provides geojsonl, along with a small but growing number of tools.]]></description><link>https://www.interline.io/blog/geojsonl-extracts/</link><guid isPermaLink="false">67253e7db62841000136ec47</guid><dc:creator><![CDATA[Ian Rees]]></dc:creator><pubDate>Tue, 11 Sep 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/preview.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/preview.png" alt="GeoJSONL: An optimized format for large geographic datasets"><blockquote><p><strong>March 8, 2019</strong>: We've published a follow-up blog post: <a href="https://www.interline.io/blog/here-cli-support-geojsonl/">Even more geospatial tools supporting the &quot;GeoJSONL&quot; format</a>. Now you can stream GeoJSONL files up to HERE XYZ, down to a web browser, or between your own programs using a growing list of software libraries.</p></blockquote><p><a href="http://geojson.org/">GeoJSON</a> is a popular format for exchanging geographic data. It builds on the success and flexibility of JSON to describe structured data, and simplifies many of the more complicated aspects of traditional GIS file formats. Initially developed through an ad hoc process, GeoJSON is now formalized by <a href="https://tools.ietf.org/html/rfc7946">IETF RFC 7946</a>, with a <a href="https://datatracker.ietf.org/wg/geojson/charter/">working group</a> to guide future additions. The universality of the underlying JSON format and the <a href="https://en.wikipedia.org/wiki/World_Geodetic_System">WGS84</a> coordinate system make GeoJSON nearly ideal for web-based mapping projects.</p><p>However, very large data sets are common in GIS, and the structure of a GeoJSON file generally requires the entire file to be read into memory and decoded all at once. A large file (e.g. 1 Gb) can easily exhaust the resources of a desktop computer. While streaming JSON parsers exist, they are generally not part of standard libraries and introduce yet another dependency and complexity.</p><p>One potential solution to the problem of large JSON files is to restructure the file as an array of objects, rather than having a single root object. Because the JSON specification prohibits the newline <code>\n</code> character from being used in string literals without escaping, it opens the possibility of having one JSON object per line in a way that is compatible with both standard UNIX text-processing tools and can be read line-by-line with existing JSON parsers.</p><p>This convention has arisen independently in many contexts, and is known as <a href="http://ndjson.org/">Newline Delimited JSON</a> (<code>ndjson</code>), <a href="http://jsonlines.org/">JSON Lines</a> (<code>jsonl</code>), or <a href="https://tools.ietf.org/html/rfc8142">GeoJSON Text Sequences</a>.<a href="#footnote-1">*</a> For GeoJSON specifically, the root level <code>FeatureCollection</code> object is replaced with a simple array of features, one per line. This file can then be read line-by-line, feature-by-feature, and easily integrated with other tools that use newline-delimited records such as <a href="https://www.gnu.org/software/parallel/">GNU parallel</a>. Newline-delimited GeoJSON (GeoJSONL) has been casually <a href="https://macwright.org/2015/03/23/geojson-second-bite.html#streaming">proposed</a> several times before, and is natively supported by some tools such as the <a href="https://docs.osmcode.org/osmium/latest/osmium-export.html">Osmium</a> <code>export</code> command, Mapbox’s <a href="https://github.com/mapbox/tippecanoe">Tippecanoe</a>, and <a href="https://stedolan.github.io/jq/">jq</a>, the Swiss-army knife of JSON processors.</p><p>At Interline, our <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extract service</a> now provides these files as well as traditional GeoJSON and OSM PBF formats, with the goal of providing a simple basis for reading and filtering OSM extracts without the need for more complicated and specialized libraries. Here is an <a href="https://storage.googleapis.com/osm-extracts.interline.io/honolulu_hawaii.geojsonl">example GeoJSONL</a> file, which you can compare to <a href="https://storage.googleapis.com/osm-extracts.interline.io/honolulu_hawaii.geojson">regular GeoJSON</a>.</p><p>While the file sizes are nearly identical, reading and parsing the entire ~27 Mb GeoJSON file requires several seconds and ~240 Mb of memory. Iterating through the GeoJSONL file line-by-line takes a similar amount of time, but uses negligible amounts of memory:</p><pre><code><span><span>import json
</span></span><span><span>with open('honolulu_hawaii.geojsonl') as f:
</span></span><span><span>    for feature in f:
</span></span><span><span>        print json.loads(feature)
</span></span></code></pre><p>The above Python snippet also demonstrates that no additional dependencies are required to read the GeoJSONL file; a line iterator and parsing each row separately works well. Still, <a href="http://ndjson.org/libraries.html">libraries are available</a> for many languages that provide native support.</p><p>As mentioned above, GeoJSONL also provides excellent input for traditional text-processing tools. The following (contrived) example runs <code>jq</code> to find the frequency of <code>name</code>s for features with the <code>highway</code> tag, without loading the entire file into memory:</p><pre><code><span><span># cat honolulu_hawaii.geojsonl | jq 'select(.properties.highway!=null) | .properties.name' | sort | uniq -c | sort -n
</span></span><span><span>..</span><span>.snip.</span><span>..
</span></span><span><span>  36</span><span> &quot;State Highway 83&quot;
</span></span><span><span>  49</span><span> &quot;John A. Burns Freeway&quot;
</span></span><span><span>  59</span><span> &quot;Makakilo Drive&quot;
</span></span><span><span> 106</span><span> &quot;Farrington Highway&quot;
</span></span><span><span> 107</span><span> &quot;Kamehameha Highway&quot;
</span></span></code></pre><p>Given the increasing complexity and importance of geographic data in nearly every domain, and with increasing file sizes, GeoJSONL is useful and timely. We’re glad to now support the format in <a href="https://app.interline.io/osm_extracts/interactive_view">Interline OSM Extracts</a> and we encourage other tools to offer native support when possible.</p><hr><p><a id="footnote-1"></a></p><h2 id="notes">Notes</h2><p><sup>*</sup> Minor differences exist between <code>ndjson</code> and <code>jsonl</code>.</p><p>GeoJSON Text Sequences adds a record separator (RS) character, in addition to the newline used by <code>ndjson</code> and <code>jsonl</code>.</p><p>Based on these existing options, we recommend the use of <code>ndjson</code>, which has a <a href="https://github.com/ndjson/ndjson-spec">simple spec document</a> and allows the presence of blank lines. This supports the widest range of tools for parsing and consuming. We use <code>geojsonl</code> as the file extension, although this is just a matter of taste.</p><style>html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}</style>]]></content:encoded></item><item><title><![CDATA[Producing 200 OpenStreetMap extracts in 35 minutes using a scalable data workflow]]></title><description><![CDATA[Interline now offers OSM Extracts, a service enabling software developers and GIS professionals to download chunks of OpenStreetMap data for 200 major cities and regions around the world.]]></description><link>https://www.interline.io/blog/scaling-openstreetmap-data-workflows/</link><guid isPermaLink="false">67255665b62841000136edcb</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 25 May 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/argo-workflow-screenshot.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/argo-workflow-screenshot.png" alt="Producing 200 OpenStreetMap extracts in 35 minutes using a scalable data workflow"><p>Interline now offers <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a>, a service enabling software developers and GIS professionals to download chunks of OpenStreetMap data for 200 major cities and regions around the world.</p><p>OSM Extracts is simple. It’s a 3 step process that you can run yourself. We’ll describe these steps, as well as how we run those 3 steps for all 200 extracts in parallel on Interline’s cloud infrastructure.</p><figure><img src="https://www.interline.io/content/images/2024/11/osm-extracts-screenshot.png" alt="screenshot of OSM Extracts website" width="1177" height="607"><figcaption><p>A screenshot of <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts by Interline</a>.</p></figcaption></figure><h2 id="creating-osm-extracts-in-3-steps">Creating OSM Extracts in 3 steps</h2><p>Every day at Interline our servers download the latest updates from <a href="https://www.openstreetmap.org/">OpenStreetMap</a>, update our local copies of the planet file, and process this data to generate a variety of geospatial data products. Reliably updating a planet file requires a bit of legwork behind the scenes, which we have encapsulated as helper scripts in our <a href="https://github.com/interline-io/planetutils">PlanetUtils</a> library. This simplifies the process down to 3 steps:</p><ol><li>Run <code>osm_planet_update</code> to download the most recent OSM planet file (released weekly at <a href="https://planet.openstreetmap.org/">planet.openstreetmap.org</a> and other mirrors) and then apply minutely “diffs” to update the planet file to the current point in time.</li><li>Fetch a copy of <a href="https://github.com/interline-io/osm-extracts/blob/master/cities.json">cities.json</a> file from GitHub. This GeoJSON file contains polygons defining the boundaries of each extract, and is a continuation of the extract definitions created by over 100 contributors for the Mapzen Metro Extracts service. The file is open to additions and revisions by all.</li><li>Run <code>osm_planet_extract</code>, specifying the local copy of the cities.json file, to generate extracts for each of the cities/regions.</li></ol><p>Behind the scenes, PlanetUtils handles calls to <code>osmosis</code> and <code>osmconvert</code> to update the planet and generate the extracts. For more information on each of these commands, see the <a href="https://github.com/interline-io/planetutils/blob/master/README.md">PlanetUtils readme</a>, or install the package yourself using Homebrew or Docker.</p><p>This workflow is simple but slow. Updating the planet and generating all 200 extracts on a single machine can take more than a day. In addition to the extracts, the updated planet is the starting point for additional data products, such as our <a href="https://www.interline.io/valhalla/tilepacks/">Valhalla Tilepacks</a>. Valhalla Tilepacks combine the OSM planet file with 1.6Tb of elevation data to produce downloadable data for organizations to run their own instances of the <a href="https://www.interline.io/valhalla/">Valhalla routing engine</a>. All of this heavy lifting requires a substantial mix of computing resources. Parallel workflows to the rescue!</p><h2 id="managing-our-osm-extracts-workflow">Managing our OSM Extracts workflow</h2><figure><img src="https://www.interline.io/content/images/2024/11/argo-workflow-screenshot-1.png" alt="screenshot of Argo workflow dashboard" width="1378" height="1462"><figcaption><p>OSM Extracts and Valhalla Tilepacks are all generated using an Argo workflow. This is screenshot from Argo's management dashboard. To simplify the image, only 2 extract jobs are shown.</p></figcaption></figure><p>At Interline, we have a great appreciation for <a href="https://kubernetes.io/">Kubernetes</a> which powers much of our infrastructure. Kubernetes is ideal for running highly available services and managing complex resources. We wanted our data workflow system to leverage this infrastructure and introduce as little additional complexity as possible. With these constraints, we found that <a href="https://applatix.com/open-source/argo/">Argo Workflow Manager</a> was a great fit for our needs:</p><ul><li>Each Argo workflow step is a <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">Kubernetes pod</a> and <a href="https://www.docker.com/what-container">Docker container</a>, simplifying resource requests and access to secrets (e.g., passwords and access keys for external services).</li><li>Workflows can be described as a directed graph, with an easy syntax for specifying dependencies between steps.</li><li>Argo provides a flexible method for specifying output files and parameters that can be fed into subsequent steps.</li><li>Parallel workflows are well supported, and one step can “fan out” to a number of parallel tasks (as you can see in the diagram above).</li></ul><p>Our Argo workflow resembles the simple 3-step process with a few key adjustments:</p><ol><li>Update our local planet using <code>osm_planet_update</code> and copy to our cloud storage bucket.</li><li>Fetch the <code>cities.json</code> file, and divide the GeoJSON features into smaller chunks of about 8 extracts each.</li><li>Run each chunk as a parallel task to quickly generate all 200 extracts.</li></ol><p>Because each Argo task is defined as a Kubernetes pod, we have very fine-grained control over the resources allocated to each task. The Kubernetes scheduler is very effective at auto-scaling your cluster based, quickly increasing cluster size to run all tasks in parallel, and just as importantly, reducing the cluster size when the tasks are complete. Additionally, because some tasks are CPU intensive and others are memory intensive, we define multiple node pools to ensure a high utilization of cluster resources.</p><p>Currently, we run about 25 tasks of 8 extracts each. With each task requesting an 8 CPU node, generating all 200 extracts in parallel takes about 35 minutes. This workflow also allows us to run multiple data pipelines together, sharing common outputs and reducing the amount of work that has to be duplicated. For instance, the updated planet file generated in the first step above is also used as the input to our Valhalla workflow.</p><h2 id="next-steps-in-the-workflow">Next steps in the workflow</h2><p>Our strategy for working with OpenStreetMap and other data sources is “snout to tail” — that is, we look for opportunities to turn each piece of a giant data download into a useful “meal” for users. We’re working on additional inputs and outputs to Interline’s workflow. Tuning the workflow is also always a work in process. Each week, we find more ways to improve the performance of a step. We look forward to sharing more updates on our workflow in the future.</p><p>In the meantime, we’d like to invite you to add a few next steps to your “professional workflow”:</p><ul><li>If you work with OpenStreetMap data, sign up for the free <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> developer preview.</li><li>If your organization needs to deploy a worldwide routing engine on premise or its in own cloud account, evaluate <a href="https://www.interline.io/valhalla/tilepacks/">Valhalla Tilepacks</a>.</li><li>If your organization runs its own custom data workflow and needs help consider these or other workflow approaches, contact us to discuss Interline’s <a href="https://www.interline.io/consulting/">consulting service</a>. We’re glad to share our experiences with Argo, Kubernetes, and related technologies.</li></ul>]]></content:encoded></item></channel></rss>