<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0"><channel><title><![CDATA[Ian Rees — Interline Technologies]]></title><description><![CDATA[We're a product and consulting business that helps organizations understand and improve transportation networks, digitally.]]></description><link>https://www.interline.io/</link><image><url>https://www.interline.io/favicon.ico</url><title>Ian Rees — Interline Technologies</title><link>https://www.interline.io/</link></image><lastBuildDate>Mon, 25 Nov 2024 19:36:52 GMT</lastBuildDate><atom:link href="https://www.interline.io/blog/author/ian/rss.xml" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[How current is that schedule?]]></title><description><![CDATA[The GTFS Best Practices guide recommends publishing data for the current data plus at least 7 days in the future. We've used the Transitland archives to assess how many feeds meet this goal.]]></description><link>https://www.interline.io/blog/how-current-is-that-schedule/</link><guid isPermaLink="false">6740e5ec598b590001b04997</guid><dc:creator><![CDATA[Ian Rees]]></dc:creator><pubDate>Mon, 25 Nov 2024 19:36:52 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/unnamed--1--1-1.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/unnamed--1--1-1.png" alt="How current is that schedule?"><blockquote><p>This blog post is based on a presentation by Ian Rees and a panel discussion along with representatives from Google, Transit app, and MobilityData regarding &quot;<em>How to Make Sure Transit Riders Get New Schedules from GTFS Static Datasets on Time?</em>&quot; at the <a href="https://mobilitydata.org/the-2024-international-mobility-data-summit-new/">2024 International Mobility Data Summit</a>.</p></blockquote><p><a href="https://www.interline.io/transitland/">Transitland</a> monitors schedule data for thousands of public transit operators. This data is collected and archived — currently at a pace of tens of thousands of updates per year:</p><figure><img src="https://www.interline.io/content/images/2024/11/unnamed.png" alt="" width="1640" height="1012"><figcaption><p>The number of <a href="https://www.transit.land/documentation/concepts/static-gtfs-feed-versions/">feed versions</a> in the Transitland archive continues to grow year by year.</p></figcaption></figure><p>We rely on operators to set their own policies for updates to their schedule data, and the time period covered by that schedule data. Operators may publish data that contains scheduled service for the next week, the next month, or even the next year. In some instances, operators publish data that only contains service that begins at a future date. This is a critical, and often hidden, factor in how useful the data is to riders and other data consumers.</p><h2 id="the-7-day-rule">The &quot;7-day rule&quot;</h2><p>Data that changes frequently or data that only contains schedules for the next few days struggles against an important constraint: Data consumers might require many hours or days before data updates can be processed and available. (For example, to rebuild a routing graph used in trip-planning software.) These constraints mean service changes may be in effect, on the ground, before these changes can be communicated back to riders, via apps and websites.</p><p>Schedules that contain less than a week of future scheduled service can also force consumers to make (often faulty) assumptions about trips on future dates.</p><p>Currently, the <a href="https://gtfs.org/documentation/schedule/schedule-best-practices/">GTFS Best Practices</a> guide recommends publishing data for the current data plus at least 7 days in the future; we'll call this the &quot;7-day rule.&quot;</p><figure><img src="https://www.interline.io/content/images/2024/11/unnamed--1--1.png" alt="" width="1640" height="1012"><figcaption><p>The percentage of feed versions in Transitland's archive by year (on the x-asis) that have 7 days, 30 days, or 90 days of forward-looking schedule coverage as of when they were fetched by Transitland.</p></figcaption></figure><p>Fortunately, this is a mostly &quot;good news&quot; situation.</p><h2 id="using-transitlands-feed-archive-to-assess-update-frequency">Using Transitland's feed archive to assess update frequency</h2><p>Averaged across all transit operators in Transitland, about 95% of static schedule updates follow the 7-day rule. This number remains about 90% even when expanding the future service window to 30 days.</p><p>It's also a good picture when disaggregating and looking at the individual operator level: The vast majority of operators have 100% of schedule updates following the 7-day rule.</p><p>The most common exceptions are:</p><ol><li>operators that have a tendency to publish data that begins in the future, with no scheduled service on the date of publication</li><li>operators that fail to update the values in <code>feed_info.txt</code> which explicitly set the range of valid service dates and remove guesswork and ambiguity for the data consumer.</li></ol><p>Even if these exceptions apply to comparatively few feeds, it's still unfortunate if they occur in feeds that are important to a data consumer's use-case (e.g., to provide up-to-date trip-planning for travelers on an important transit operator).</p><h2 id="even-more-carefully-characterizing-updates">Even more carefully characterizing updates</h2><p>This provides an optimistic baseline for the current state of GTFS updates. For an even more holistic picture, we also need to incorporate a few additional metrics:</p><ul><li>Percentage of days for each operator where the data is &quot;stale&quot;, that is, outside the scheduled service window for the last published data.</li><li>The number of operators that publish data on a very frequent basis that assumes consumers have a fast turn-around process, and the characteristics of these updates.</li></ul><p>These questions will be explored in a future blog post.</p><blockquote><p><strong>The take-away from this blog post for transit data producers</strong>: Please check to ensure that your GTFS feed always includes service for the current point-in-time when it is published, and include at least 7 days of forward-looking schedule coverage (if not 30 or 90 days).</p></blockquote>]]></content:encoded></item><item><title><![CDATA[Even more geospatial tools supporting the "GeoJSONL" format]]></title><description><![CDATA["GeoJSONL" has the easy readability of a GeoJSON file without the the performance limitations of the format. Last year we started producing GeoJSONL files from our OSM Extracts service. Now a growing list of long-existing and brand-new geospatial tools.]]></description><link>https://www.interline.io/blog/here-cli-support-geojsonl/</link><guid isPermaLink="false">6725410eb62841000136ec69</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 08 Mar 2019 08:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/download-geojsonl-extract.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/download-geojsonl-extract.png" alt="Even more geospatial tools supporting the &quot;GeoJSONL&quot; format"><p>Last year we expanded our <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> service to produce what we call GeoJSONL. This is a data format that goes by many names: newline-delimited GeoJSON, line-oriented GeoJSON, GeoJSONSeq, or GeoJSON Text Sequences. Whatever its name, the format enables effficient reading and efficient writing of <a href="https://en.wikipedia.org/wiki/GIS_file_formats#Vector">vector geo-data</a> features.</p><p>For more background, see our <a href="https://www.interline.io/blog/geojsonl-extracts/">previous blog post on GeoJSONL</a> and see this new <a href="https://stevage.github.io/ndgeojson/">overview of the format’s many names and its defining properties</a>.</p><p>Over the last year, there’s been even more progress on tools for writing and reading GeoJSONL (or whatever your preferred name is for this format).</p><h2 id="geojsonl-support-in-here-xyz">GeoJSONL support in HERE XYZ</h2><p>HERE XYZ now supports GeoJSONL ingest.</p><p>Don’t know what HERE XYZ is? Here’s how we summarized the platform in a <a href="https://www.interline.io/blog/here-xyz-open-geo-data/">blog post</a> last year:</p><blockquote><p>HERE XYZ provides the full set of components that &quot;modern&quot; geo developers have come to expect: GeoJSON input and output, &quot;slippy map&quot; libraries, a RESTful API, a JavaScript SDK, a cross-platform CLI, base maps and styles. As we've been experimenting with the HERE XYZ beta, we've found it provides some unique combinations that make it especially well suited to combining open geodata APIs — especially in large quantities.</p></blockquote><p>Using the latest version of the <a href="https://www.npmjs.com/package/@here/cli">HERE CLI</a>, you can parse and upload GeoJSONL files to the HERE XYZ platform.</p><p>Here’s an update of our <a href="https://interline-io.github.io/here-xyz-tutorials/tutorial-1/">“Use Crowdsourced Data from Interline OSM Extracts in an XYZ Web Map” tutorial</a> that uses the GeoJSONL format.</p><p>Download a GeoJSONL extract from OSM Extracts:</p><figure><img src="https://www.interline.io/content/images/2024/11/download-geojsonl-extract-1.png" alt="terminal showing usage of HERE CLI to upload OSM Extract in GeoJSONL format" width="2000" height="1085"></figure><p>Upload to XYZ and set tags:</p><figure><img src="https://www.interline.io/content/images/2024/11/here-cli-upload-geojsonl.png" alt="screenshot of a terminal window" width="1688" height="502"></figure><p>Previously you might hit memory limits on your computer if you tried to upload an extract for a big metro region. Now, thanks to GeoJSONL’s efficient parsing, you should be able to upload any extract of any size!</p><h2 id="the-growing-toolchain">The growing toolchain</h2><p>There’s a long history of tools that work with this format, and it’s been growing over the last year. Here are some more highlights:</p><p><strong>Command line</strong> On the command line, read <code>.geojsonl</code> and <code>.geojsons</code> files using GDAL’s <a href="https://gdal.org/drivers/vector/geojsonseq.html"><code>drv_geojsonseq</code></a>.</p><p><strong>Desktop GIS</strong> GDAL also powers <a href="https://www.qgis.org/">QGIS</a>, so now you can load <code>.geojsonl</code> and <code>.geojsons</code> files into our favorite desktop GIS sofware.</p><p><strong>Parse in Python</strong>: <a href="https://github.com/Toblerity/Fiona">Fiona</a> is a Python library that wraps GDAL and provides a “neat and nimble” interface. The next release will include <a href="https://github.com/Toblerity/Fiona/issues/720">support</a> for <code>drv_geojsonseq</code>.<a href="https://www.interline.io/blog/here-cli-support-geojsonl/#footnote-1">*</a></p><p><strong>Statistics and data science in R</strong> If you’re loading vector geo-data into R, try the <a href="https://cran.r-project.org/package=geojson">geojson</a> package on CRAN. See these <a href="https://github.com/ropensci/geojson#newline-delimited-geojson">instructions</a> for how to load a <code>.geojsonl</code> file directly from Interline OSM Extracts.</p><p><strong>Convert from a shapefile</strong> Install the <a href="https://www.npmjs.com/package/shapefile">shapefile NPM package</a> and run <code>shp2json --newline-delimited</code> to convert a shapefile to GeoJSONL. See these <a href="https://github.com/mbostock/shapefile#shp2json_newline_delimited">instructions</a>.</p><p><strong>Convert to vector tiles</strong> <a href="https://github.com/mapbox/tippecanoe/">Tippecanoe</a> can take GeoJSONL as input and turn it in to <a href="https://github.com/mapbox/awesome-vector-tiles">MVT vector tiles</a>. It also provides its own <code>tippecanoe-json-tool</code> command to convert a standard GeoJSON file to GeoJSONL format (see <a href="https://github.com/mapbox/tippecanoe/#tippecanoe-json-tool">these instructions</a>).</p><p><strong>Stream in to a web browser</strong> Watch a GeoJSONL file stream into a web in this <a href="https://observablehq.com/@bcamper/streaming-geojsonl">interactive JavaScript notebook</a>.<a href="https://www.interline.io/blog/here-cli-support-geojsonl/#footnote-2">†</a> If you have an Interline OSM Extracts API key and an Observable account, try <a href="https://observablehq.com/@drewda/streaming-geojsonl-from-your-choice-of-openstreetmap-extr">this notebook</a> to select from any of the 200+ cities and watch the stream arrive:</p><figure><img src="https://www.interline.io/content/images/2024/11/streaming-geojsonl.jpg" alt="" width="600" height="547"></figure><h2 id="when-to-try-geojsonl">When to try GeoJSONL</h2><p>If you like GeoJSON but find your data files are too big to hold in memory, give one of these tools a try.</p><p>If you need to run a bunch of vector geo-data through a <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> operation that, give this format a try. (It’s easy to divide a GeoJSONL file in to chunks before parsing its exact contents.)</p><p>If you want to download <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> from Interline and explore even the largest of extracts using HERE XYZ, give this a try.</p><hr><h3 id="notes">Notes</h3><p><a id="footnote-1"></a></p><p><sup>*</sup> Shoutout to <a href="https://github.com/sgillies">Sean Gillies</a>, the lead creator of Fiona and instigator of <a href="https://tools.ietf.org/html/rfc8142">RFC 8142 - GeoJSON Text Sequences</a>. We may have differing views on the <a href="https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text">record separator character</a>. Regardless, his packages like Fiona and <a href="https://rasterio.readthedocs.io/">Rasterio</a> are fabulous and we’re happy users of them.</p><p><a id="footnote-2"></a></p><p><sup>†</sup> Shoutout to <a href="http://vector.io/">Brett Camper</a>, the author of this Observable notebook and a fellow alum of <a href="https://mapzen.com/">Mapzen</a>.</p>]]></content:encoded></item><item><title><![CDATA[Transitland Version 2]]></title><description><![CDATA[What first started as two simple, separate, little experiments has grown into the seeds of a major revision of the entire Transitland platform: Transitland Version 2.]]></description><link>https://www.interline.io/blog/tlv2/</link><guid isPermaLink="false">67255876b62841000136ee13</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Tue, 08 Jan 2019 08:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/tlv2-architecture-diagram.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/tlv2-architecture-diagram.png" alt="Transitland Version 2"><blockquote><p><strong>March 4, 2021</strong>: Here's a more recent blog post <a href="https://www.interline.io/blog/transitland-v2-search/">demonstrating Transitland v2's new ability to search across transit operators, feeds, and routes</a>.</p></blockquote><blockquote><p><strong>February 25, 2021</strong>: Here's a more recent blog post <a href="https://www.interline.io/blog/transitland-v2-vector-map-tiles/">demonstrating Transitland v2's new global transit map and vector tiles</a>.</p></blockquote><p>It's been a quiet year on this blog, and we're overdue to share all the work that's been going on behind the scenes since the <a href="https://www.interline.io/blog/transitland-continues/">last post</a>. What first started as two simple, separate, little experiments has grown into the seeds of a major revision of the entire Transitland platform: Transitland Version 2.</p><h2 id="tlv2-functionality-and-goals">Tlv2 Functionality and Goals</h2><p><em>Tlv2</em> (as we've come to call this new version) provides continuity on the most used parts of the Transitland platform. Tlv2 will still provide a canonical service at transit.land that includes a Feed Registry of open GTFS data feeds from around the world, a queryable API, and a range of user interfaces. Tlv2, like Tlv1, will be powered by open-source components. Tlv2's API will continue to be accessible without charge, and most of the Tlv1 API will continue to be available during the transition. (As before, we will enforce rate limits to ensure no one user hogs too many server resources; more on how rate limits will change below.)</p><p>What we are changing in Tlv2 is in support of three goals:</p><ul><li>Higher Performance</li><li>More Modular Architecture</li><li>Distributed Deployments</li></ul><h2 id="goal-1-higher-performance">Goal #1: Higher Performance</h2><p>Tlv1 successfully parses and imports GTFS feeds from thousands of transit agencies. However its &quot;FeedEater&quot; component slows down or halts on a handful of the very larger GTFS feeds — typically single feeds that cover entire countries. This has always been a constraint on Tlv1.</p><p>Interline's new <code>transitland-lib</code> (tlib) library solves this problem. Transitland-lib is a new transit data library written in the <a href="https://golang.org/">Go programming language</a>, which provides finer-grained control of memory usage, as well as much easier patterns for concurrency and parallelization of complex tasks. Transitland-lib replaces replaces FeedEater, and will be used for feed validation, pre-processing, and importing data into the Tlv2 database. Transitland-lib is efficient enough to handle the largest feeds in a small memory footprint, and is fast enough to encourage interactive use; imports that took hours with FeedEater are completed in minutes.</p><p>Transitland-lib began as an experiment by Ian Rees over the last winter holidays, and it now complete enough to power multiple Interline projects, from the San Francisco Bay Area Regional GTFS Feed (which merges ~30 feeds into one) to a transit-to-routing-engine pipeline that Interline built for another client (which covers transit service for an entire country!). We look forward to releasing transitland-lib under an open-source license shortly.</p><p>Transitland's most demanding API users have also hit performance limitations over the years. As part of Tlv2, we'll be providing bulk data dumps that give users another option. Use the Tlv2 API to explore data or power a small web app; use Tlv2 data extracts to power a large analysis, a giant web app, or a country-wide routing engine.</p><h2 id="goal-2-more-modular-architecture">Goal #2: More Modular Architecture</h2><p>Tlv1 has succeeded in part due to its &quot;monolith&quot; architecture. FeedEater, the API, and administrative operations all live within the same <a href="https://www.transit.land/documentation/datastore/">Transitland Datastore</a> codebase. This has allowed for efficient reuse of data models, helpers, and test code. The single Datastore codebase is also straightforward to deploy and configure for the canonical transit.land site. The Datastore codebase can be reused — some have in fact redeployed it for their own regional or internal GTFS management needs—but customizing and redeploying the Datastore is not straightforward.</p><p>For Tlv2, we're breaking the Transitland Datastore into modular components. Some components will be in Go (data processing), some will remain in Ruby (general back-end) and JavaScript (user interfaces). The canonical transit.land deployment will continue to use all. Other deployments by Interline, our partners, or open-source users may only use a subset. This more modular architecture will also allow deployments that connect Tlv2 components to internal data pipelines, routing engines, and other tools that may not be relevant to the canonical transit.land deployment.</p><p>While the software components of Tlv2 will be more modular, they will still be glued together using the data schemes developed for Transitland, including <a href="https://www.transit.land/documentation/concepts/onestop-id-scheme">Onestop IDs</a> and a common feed format spec (described below). Additionally, we will be sharing our database schema in versions appropriate for PostgreSQL and SQLite to help simplify additional application development, including outside of the canonical transit.land.</p><h2 id="goal-3-distributed-deployments">Goal #3: Distributed Deployments</h2><p>The Feed Registry at transit.land will continue to be our canonical list of GTFS feeds. However, some deployments of Tlv2 components may want to use just a subset of the Feed Registry or a private list of feeds. We also want to open up the canonical Feed Registry for more collaborative tending, beyond just staff from Interline and Trillium Transit.</p><p>The second of the two experiments that has led to Tlv2 is called the <a href="https://github.com/transitland/distributed-mobility-feed-registry">Distributed Mobility Feed Registry</a>. DMFR (for short) is a standard way of representing a list of GTFS feeds as one or more JSON files. It also currently supports GTFS-RT, and we expect it to extend to GBFS (bike rental) and MDS (e-scooter rental) in the future.</p><p>(Some history: This experiment actually started way back in the early tests of Transitland, before the fully built out Tlv1 monolith. It's still <a href="https://github.com/transitland/transitland-datastore">archived</a> on GitHub. Now we've come full circle to revisit and build out this approach.)</p><p>Soon the <code>transitland/distributed-mobility-feed-registry</code> repository on GitHub will become the source of truth for the Transitland Feed Registry of public open data feeds. We'll accept pull requests against the JSON files to add or edit feeds. This should substantially lower the burden for outside contributors to contribute new feeds and help manage existing records. However, this will be just be one repository of DMFR files. In effect we're blowing up the single Transitland Feed Registry and inviting the creation of many.</p><p>Transitland-lib can manage a datastore using one or more DMFR files, meaning that the canonical transit.land, distributed instances of Tlv2, and temporary interactive sessions with transitland-lib on the command line can all share the same representation for mobility data feeds. At Interline, we've gone one step further and also use DMFR files to configure some of our routing engines and any other workflow that takes two or more mobility data feeds as input. We welcome other consumers or transformers of GTFS, GTFS-RT, and related data to try a DMFR file for their own needs.</p><h2 id="tlv2-architecture-diagram">Tlv2 Architecture Diagram</h2><p>Now that we've introduced and discussed the new components of Tlv2, here's a diagram of how they connect:</p><figure><img src="https://www.interline.io/content/transit-land/tlv2/tlv2-architecture-diagram.png" alt="Tlv2 architecture diagram" width="768" height="942"></figure><h2 id="gradual-deprecation-of-tlv1">Gradual Deprecation of Tlv1</h2><p>Tlv1 requires a fair amount of attention and resources to maintain. While developing Tlv2, we've already had to split our attention. As we shift our focus to deploying and supporting Tlv2, we'll be gradually deprecating Tlv1. Don't worry — it won't be too quickly.</p><p><strong>Step 1: Tlv2 API Preview</strong> The initial release of Tlv2 will include a preview of a new <a href="https://graphql.org/">GraphQL</a> API that provides flexible, structured access to transit data. All new data will be available through this API from the start. API access will require a key. Sign-up will be free. We will institute tight rate limits at first and increase them over time, as we better understand the type and complexity of queries that users run.</p><p><strong>Step 2: Deprecate Tlv1 Schedule API</strong> Immediately upon Tlv2 release, we will stop importing new <a href="https://www.transit.land/documentation/datastore/schedules.html">schedule data</a> into Tlv1. The <code>ScheduleStopPair</code> data model is the most expensive part of Tlv1 to run and maintain. We'll first stop imports, keeping the API available against the last imported version. We'll share tutorials on how to switch to the Tlv2 GraphQL API to access schedules. Then, after sufficient notice, we'll empty old Tlv1 schedule data and remove that API endpoint.</p><p><strong>Step 3: Other Tlv1 Entity API Endpoints</strong> While the Tlv2 GraphQL API will make a number of complex queries much simpler than with the current Tlv1 API, it may take a while for us to make all Tlv1 query options available in the Tlv2 API. During this time, we'll try to continue to power the Tlv1 APIs (with the exception of the schedules endpoint). We'll recommend that you build any new applications on the Tlv2 API, in case we do have to announce any specific incompatibilities during the Tlv1-to-Tlv2 API transition.</p><p>We'll post updates to this site. Or watch <a href="https://twitter.com/transitland">@transitland</a> and <a href="https://twitter.com/interline_io">@interline_io</a> for updates on Twitter.</p><h2 id="how-you-can-help">How You Can Help</h2><p>Here are ways you can help with the rollout of Tlv2:</p><ul><li>Provide your input on Tlv2 by taking our <a href="https://docs.google.com/forms/d/e/1FAIpQLSdmhW28zcP1kBMBkmelAt7OQGqKjRcoILjsbZvGXLsGfmFNGw/viewform">survey and signing up for the beta tester list</a>.</li><li>Comment on the <a href="https://funding.communitybridge.org/projects/transitland">DMFR GitHub repository</a>. Contribute edits once we migrate over the Feed Registry to DMFR files.</li><li>If you work for a GTFS producing transit operator or mobility provider, <a href="mailto:info@interline.io">contact Interline</a> to learn more about &quot;Tlv2 on premise&quot; and regional GTFS feeds. It's consulting projects like this that help us serve both mobility providers and further the development of transitland-lib.</li><li>Contribute financially to the upkeep and growth of Tlv2 through the Linux Foundation's <a href="https://funding.communitybridge.org/projects/transitland">CommunityBridge donation page for Transitland</a>. These contributions go directly toward upkeep of the canonical transit.land.</li></ul>]]></content:encoded></item><item><title><![CDATA[Bringing Back the Spirit of Map Mashups: Combine Open Geodata APIs using HERE XYZ]]></title><description><![CDATA[Interline and HERE are collaborating to make Interline's open data services like OSM Extracts and the Transitland API easy to combine, mix, filter, analyze, and visualize using the new HERE XYZ platform. ]]></description><link>https://www.interline.io/blog/here-xyz-open-geo-data/</link><guid isPermaLink="false">6725453eb62841000136ecbc</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Mon, 01 Oct 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/xyz-tutorial-1-animation.jpg" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/xyz-tutorial-1-animation.jpg" alt="Bringing Back the Spirit of Map Mashups: Combine Open Geodata APIs using HERE XYZ"><p>At Interline, we spend much of our time with open geodata APIs. We create our own open geodata APIs (like <a href="https://www.transit.land/">Transitland</a>) and we use others (for example, whenever we need to throw a bunch of GeoJSON on a base map). At their best, open geodata APIs are fun — you get to combine pieces together to create something new. Then again, public APIs can have many limitations — combine together a few and you can end up with the worst of all worlds: duplicate data, misregistered overlays, a slow user experience, incompatible data formats… With all these mixed experiences in our minds, we’ve been playing with the initial beta of <a href="http://explore.xyz.here.com/">HERE XYZ</a>. It’s a brand new developer service from one of the oldest names in geodata.</p><p>HERE XYZ provides the full set of components that “modern” geo developers have come to expect: GeoJSON input and output, “slippy map” libraries, a RESTful API, a JavaScript SDK, a cross-platform CLI, base maps and styles. As we’ve been experimenting with the HERE XYZ beta, we’ve found it provides some unique combinations that make it especially well suited to combining open geodata APIs — especially in large quantities.</p><p>Combining geodata is nothing new. <a href="http://ladprofile.weebly.com/ian-mcharg-1920-2001.html">Ian McHarg</a>, a prominent environmental designer, printed maps on transparent plastic sheets and overlaid them atop each other, inspiring the idea of layers in GIS (<a href="https://en.wikipedia.org/wiki/Geographic_information_system">geographic information systems</a>):</p><figure><img src="https://www.interline.io/content/images/2024/11/ian-mcharg.jpg" alt="photo of Ian McHarg" width="1100" height="742"></figure><p><em>Image credit: University of Pennsylvania</em></p><p>More recently, Google Maps brought us <em>mashups</em>. At first, it involved hacking the Google Maps JavaScript code in order to overlay <a href="https://www.nytimes.com/2005/12/11/magazine/doityourself-cartography.html">crimes in Chicago</a> and <a href="http://www.housingmaps.com/">Craigslist house listings</a>:</p><figure><img src="https://www.interline.io/content/images/2024/11/housingmaps2.png" alt="screenshot of old housingmaps.com" width="1088" height="684"></figure><p><em>Image credit: housingmaps.com</em></p><p>Neither hacking nor reverse-engineering are required any more to combine multiple sources of geodata in a web browser. Companies like Google, Esri, Mapbox, Carto — also a now defunct start-up where we previously worked, called Mapzen — all offer accessible APIs for creating web maps. We’ve also been freed of many of the constraints on dataset size in 2005 mashups. Original mashups involved drawing points and other geographic features on top of base-map images, with a low limit on the number of these vector features that could be drawn at once. More recent techniques send both the base map and overlays as vector tiles. Instead of an image, a vector tile is effectively a square of data, which can be pre-generated by a server. Many vector tiles are then loaded and drawn by your web browser simultaneously.</p><p>We’ve left behind the constraints of mashups — and it also feels like we’ve forgotten the fun of mashups. Vector tiles help greatly with performance, but often add complexity to the overall set of steps required to combine together existing and new data sources on a web map or another type of visualization.</p><p>Our experiments with the HERE CLI and the XYZ API are bringing back that creative spirit of 2005 mashups. The XYZ platform generates vector tiles near instantly for any new imported data sources. The HERE CLI and XYZ API provide a range of ways to import GeoJSON files, connect to APIs, or stream in from other CLI tools. It’s the best of 2005 and 2018. If you use the SDKs to style your web maps to look like the transparency overlays in Ian McHarg’s book <em>Design With Nature</em>, it’s the best of 1969 proto-GIS, too.</p><p>We’re collaborating with the HERE XYZ team to turn our experiments into tutorials that demonstrate how Interline’s open geodata services can be combined, mixed, filtered, analyzed, and visualized. In the first tutorial, you’ll fetch and filter data from <a href="https://app.interline.io/osm_extracts/interactive_view">Interline OSM Extracts</a>:</p><figure><img src="https://www.interline.io/content/images/2024/11/xyz-tutorial-1-animation-2.jpg" alt="" width="918" height="756"></figure><p>In the second tutorial, you’ll query the <a href="https://www.transit.land/">Transitland</a> API repeatedly to build up an analysis and a web map of just not subway locations but also the frequency of how often each subway line operates. (Especially when working with geodata for transportation applications, both space and time matter.) We’ll show you how to do this for the entire planet:</p><figure><img src="https://www.interline.io/content/images/2024/11/xyz-tutorial-2-animation-3.jpg" alt="" width="800" height="526"></figure><p>The sample projects and tutorials take intermediate and advanced users on a tour through some of the unique features of XYZ and the HERE CLI:</p><ul><li>filtering Interline OSM Extracts using OpenStreetMap tags</li><li>writing scripts that stream geodata from external APIs into an XYZ space using the HERE CLI</li><li>using multiple API queries to aggregate together properties about the same features (identified by unique IDs, such as <a href="https://www.transit.land/documentation/concepts/onestop-id-scheme/">Transitland Onestop IDs</a>)</li><li>quickly building global tile sets</li></ul><p>We’re experimenting with additional features, for creating a new generation of mashups, and we look forward to sharing more soon.</p><p>Now, get started with our <a href="https://www.interline.io/docs/here-xyz/">first two tutorials on combining HERE XYZ, Interline OSM Extracts, and Transitland</a>.</p>]]></content:encoded></item><item><title><![CDATA[GeoJSONL: An optimized format for large geographic datasets]]></title><description><![CDATA[GeoJSON Lines (geojsonl) is a simple, newline-delimited variant of GeoJSON that allows large datasets to be loaded with a much lower memory footprint and easily integrate with traditional text-based tools. Interline OSM Extracts now provides geojsonl, along with a small but growing number of tools.]]></description><link>https://www.interline.io/blog/geojsonl-extracts/</link><guid isPermaLink="false">67253e7db62841000136ec47</guid><dc:creator><![CDATA[Ian Rees]]></dc:creator><pubDate>Tue, 11 Sep 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/preview.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/preview.png" alt="GeoJSONL: An optimized format for large geographic datasets"><blockquote><p><strong>March 8, 2019</strong>: We've published a follow-up blog post: <a href="https://www.interline.io/blog/here-cli-support-geojsonl/">Even more geospatial tools supporting the &quot;GeoJSONL&quot; format</a>. Now you can stream GeoJSONL files up to HERE XYZ, down to a web browser, or between your own programs using a growing list of software libraries.</p></blockquote><p><a href="http://geojson.org/">GeoJSON</a> is a popular format for exchanging geographic data. It builds on the success and flexibility of JSON to describe structured data, and simplifies many of the more complicated aspects of traditional GIS file formats. Initially developed through an ad hoc process, GeoJSON is now formalized by <a href="https://tools.ietf.org/html/rfc7946">IETF RFC 7946</a>, with a <a href="https://datatracker.ietf.org/wg/geojson/charter/">working group</a> to guide future additions. The universality of the underlying JSON format and the <a href="https://en.wikipedia.org/wiki/World_Geodetic_System">WGS84</a> coordinate system make GeoJSON nearly ideal for web-based mapping projects.</p><p>However, very large data sets are common in GIS, and the structure of a GeoJSON file generally requires the entire file to be read into memory and decoded all at once. A large file (e.g. 1 Gb) can easily exhaust the resources of a desktop computer. While streaming JSON parsers exist, they are generally not part of standard libraries and introduce yet another dependency and complexity.</p><p>One potential solution to the problem of large JSON files is to restructure the file as an array of objects, rather than having a single root object. Because the JSON specification prohibits the newline <code>\n</code> character from being used in string literals without escaping, it opens the possibility of having one JSON object per line in a way that is compatible with both standard UNIX text-processing tools and can be read line-by-line with existing JSON parsers.</p><p>This convention has arisen independently in many contexts, and is known as <a href="http://ndjson.org/">Newline Delimited JSON</a> (<code>ndjson</code>), <a href="http://jsonlines.org/">JSON Lines</a> (<code>jsonl</code>), or <a href="https://tools.ietf.org/html/rfc8142">GeoJSON Text Sequences</a>.<a href="#footnote-1">*</a> For GeoJSON specifically, the root level <code>FeatureCollection</code> object is replaced with a simple array of features, one per line. This file can then be read line-by-line, feature-by-feature, and easily integrated with other tools that use newline-delimited records such as <a href="https://www.gnu.org/software/parallel/">GNU parallel</a>. Newline-delimited GeoJSON (GeoJSONL) has been casually <a href="https://macwright.org/2015/03/23/geojson-second-bite.html#streaming">proposed</a> several times before, and is natively supported by some tools such as the <a href="https://docs.osmcode.org/osmium/latest/osmium-export.html">Osmium</a> <code>export</code> command, Mapbox’s <a href="https://github.com/mapbox/tippecanoe">Tippecanoe</a>, and <a href="https://stedolan.github.io/jq/">jq</a>, the Swiss-army knife of JSON processors.</p><p>At Interline, our <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extract service</a> now provides these files as well as traditional GeoJSON and OSM PBF formats, with the goal of providing a simple basis for reading and filtering OSM extracts without the need for more complicated and specialized libraries. Here is an <a href="https://storage.googleapis.com/osm-extracts.interline.io/honolulu_hawaii.geojsonl">example GeoJSONL</a> file, which you can compare to <a href="https://storage.googleapis.com/osm-extracts.interline.io/honolulu_hawaii.geojson">regular GeoJSON</a>.</p><p>While the file sizes are nearly identical, reading and parsing the entire ~27 Mb GeoJSON file requires several seconds and ~240 Mb of memory. Iterating through the GeoJSONL file line-by-line takes a similar amount of time, but uses negligible amounts of memory:</p><pre><code><span><span>import json
</span></span><span><span>with open('honolulu_hawaii.geojsonl') as f:
</span></span><span><span>    for feature in f:
</span></span><span><span>        print json.loads(feature)
</span></span></code></pre><p>The above Python snippet also demonstrates that no additional dependencies are required to read the GeoJSONL file; a line iterator and parsing each row separately works well. Still, <a href="http://ndjson.org/libraries.html">libraries are available</a> for many languages that provide native support.</p><p>As mentioned above, GeoJSONL also provides excellent input for traditional text-processing tools. The following (contrived) example runs <code>jq</code> to find the frequency of <code>name</code>s for features with the <code>highway</code> tag, without loading the entire file into memory:</p><pre><code><span><span># cat honolulu_hawaii.geojsonl | jq 'select(.properties.highway!=null) | .properties.name' | sort | uniq -c | sort -n
</span></span><span><span>..</span><span>.snip.</span><span>..
</span></span><span><span>  36</span><span> &quot;State Highway 83&quot;
</span></span><span><span>  49</span><span> &quot;John A. Burns Freeway&quot;
</span></span><span><span>  59</span><span> &quot;Makakilo Drive&quot;
</span></span><span><span> 106</span><span> &quot;Farrington Highway&quot;
</span></span><span><span> 107</span><span> &quot;Kamehameha Highway&quot;
</span></span></code></pre><p>Given the increasing complexity and importance of geographic data in nearly every domain, and with increasing file sizes, GeoJSONL is useful and timely. We’re glad to now support the format in <a href="https://app.interline.io/osm_extracts/interactive_view">Interline OSM Extracts</a> and we encourage other tools to offer native support when possible.</p><hr><p><a id="footnote-1"></a></p><h2 id="notes">Notes</h2><p><sup>*</sup> Minor differences exist between <code>ndjson</code> and <code>jsonl</code>.</p><p>GeoJSON Text Sequences adds a record separator (RS) character, in addition to the newline used by <code>ndjson</code> and <code>jsonl</code>.</p><p>Based on these existing options, we recommend the use of <code>ndjson</code>, which has a <a href="https://github.com/ndjson/ndjson-spec">simple spec document</a> and allows the presence of blank lines. This supports the widest range of tools for parsing and consuming. We use <code>geojsonl</code> as the file extension, although this is just a matter of taste.</p><style>html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}</style>]]></content:encoded></item><item><title><![CDATA[Producing 200 OpenStreetMap extracts in 35 minutes using a scalable data workflow]]></title><description><![CDATA[Interline now offers OSM Extracts, a service enabling software developers and GIS professionals to download chunks of OpenStreetMap data for 200 major cities and regions around the world.]]></description><link>https://www.interline.io/blog/scaling-openstreetmap-data-workflows/</link><guid isPermaLink="false">67255665b62841000136edcb</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 25 May 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/argo-workflow-screenshot.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/argo-workflow-screenshot.png" alt="Producing 200 OpenStreetMap extracts in 35 minutes using a scalable data workflow"><p>Interline now offers <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a>, a service enabling software developers and GIS professionals to download chunks of OpenStreetMap data for 200 major cities and regions around the world.</p><p>OSM Extracts is simple. It’s a 3 step process that you can run yourself. We’ll describe these steps, as well as how we run those 3 steps for all 200 extracts in parallel on Interline’s cloud infrastructure.</p><figure><img src="https://www.interline.io/content/images/2024/11/osm-extracts-screenshot.png" alt="screenshot of OSM Extracts website" width="1177" height="607"><figcaption><p>A screenshot of <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts by Interline</a>.</p></figcaption></figure><h2 id="creating-osm-extracts-in-3-steps">Creating OSM Extracts in 3 steps</h2><p>Every day at Interline our servers download the latest updates from <a href="https://www.openstreetmap.org/">OpenStreetMap</a>, update our local copies of the planet file, and process this data to generate a variety of geospatial data products. Reliably updating a planet file requires a bit of legwork behind the scenes, which we have encapsulated as helper scripts in our <a href="https://github.com/interline-io/planetutils">PlanetUtils</a> library. This simplifies the process down to 3 steps:</p><ol><li>Run <code>osm_planet_update</code> to download the most recent OSM planet file (released weekly at <a href="https://planet.openstreetmap.org/">planet.openstreetmap.org</a> and other mirrors) and then apply minutely “diffs” to update the planet file to the current point in time.</li><li>Fetch a copy of <a href="https://github.com/interline-io/osm-extracts/blob/master/cities.json">cities.json</a> file from GitHub. This GeoJSON file contains polygons defining the boundaries of each extract, and is a continuation of the extract definitions created by over 100 contributors for the Mapzen Metro Extracts service. The file is open to additions and revisions by all.</li><li>Run <code>osm_planet_extract</code>, specifying the local copy of the cities.json file, to generate extracts for each of the cities/regions.</li></ol><p>Behind the scenes, PlanetUtils handles calls to <code>osmosis</code> and <code>osmconvert</code> to update the planet and generate the extracts. For more information on each of these commands, see the <a href="https://github.com/interline-io/planetutils/blob/master/README.md">PlanetUtils readme</a>, or install the package yourself using Homebrew or Docker.</p><p>This workflow is simple but slow. Updating the planet and generating all 200 extracts on a single machine can take more than a day. In addition to the extracts, the updated planet is the starting point for additional data products, such as our <a href="https://www.interline.io/valhalla/tilepacks/">Valhalla Tilepacks</a>. Valhalla Tilepacks combine the OSM planet file with 1.6Tb of elevation data to produce downloadable data for organizations to run their own instances of the <a href="https://www.interline.io/valhalla/">Valhalla routing engine</a>. All of this heavy lifting requires a substantial mix of computing resources. Parallel workflows to the rescue!</p><h2 id="managing-our-osm-extracts-workflow">Managing our OSM Extracts workflow</h2><figure><img src="https://www.interline.io/content/images/2024/11/argo-workflow-screenshot-1.png" alt="screenshot of Argo workflow dashboard" width="1378" height="1462"><figcaption><p>OSM Extracts and Valhalla Tilepacks are all generated using an Argo workflow. This is screenshot from Argo's management dashboard. To simplify the image, only 2 extract jobs are shown.</p></figcaption></figure><p>At Interline, we have a great appreciation for <a href="https://kubernetes.io/">Kubernetes</a> which powers much of our infrastructure. Kubernetes is ideal for running highly available services and managing complex resources. We wanted our data workflow system to leverage this infrastructure and introduce as little additional complexity as possible. With these constraints, we found that <a href="https://applatix.com/open-source/argo/">Argo Workflow Manager</a> was a great fit for our needs:</p><ul><li>Each Argo workflow step is a <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">Kubernetes pod</a> and <a href="https://www.docker.com/what-container">Docker container</a>, simplifying resource requests and access to secrets (e.g., passwords and access keys for external services).</li><li>Workflows can be described as a directed graph, with an easy syntax for specifying dependencies between steps.</li><li>Argo provides a flexible method for specifying output files and parameters that can be fed into subsequent steps.</li><li>Parallel workflows are well supported, and one step can “fan out” to a number of parallel tasks (as you can see in the diagram above).</li></ul><p>Our Argo workflow resembles the simple 3-step process with a few key adjustments:</p><ol><li>Update our local planet using <code>osm_planet_update</code> and copy to our cloud storage bucket.</li><li>Fetch the <code>cities.json</code> file, and divide the GeoJSON features into smaller chunks of about 8 extracts each.</li><li>Run each chunk as a parallel task to quickly generate all 200 extracts.</li></ol><p>Because each Argo task is defined as a Kubernetes pod, we have very fine-grained control over the resources allocated to each task. The Kubernetes scheduler is very effective at auto-scaling your cluster based, quickly increasing cluster size to run all tasks in parallel, and just as importantly, reducing the cluster size when the tasks are complete. Additionally, because some tasks are CPU intensive and others are memory intensive, we define multiple node pools to ensure a high utilization of cluster resources.</p><p>Currently, we run about 25 tasks of 8 extracts each. With each task requesting an 8 CPU node, generating all 200 extracts in parallel takes about 35 minutes. This workflow also allows us to run multiple data pipelines together, sharing common outputs and reducing the amount of work that has to be duplicated. For instance, the updated planet file generated in the first step above is also used as the input to our Valhalla workflow.</p><h2 id="next-steps-in-the-workflow">Next steps in the workflow</h2><p>Our strategy for working with OpenStreetMap and other data sources is “snout to tail” — that is, we look for opportunities to turn each piece of a giant data download into a useful “meal” for users. We’re working on additional inputs and outputs to Interline’s workflow. Tuning the workflow is also always a work in process. Each week, we find more ways to improve the performance of a step. We look forward to sharing more updates on our workflow in the future.</p><p>In the meantime, we’d like to invite you to add a few next steps to your “professional workflow”:</p><ul><li>If you work with OpenStreetMap data, sign up for the free <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> developer preview.</li><li>If your organization needs to deploy a worldwide routing engine on premise or its in own cloud account, evaluate <a href="https://www.interline.io/valhalla/tilepacks/">Valhalla Tilepacks</a>.</li><li>If your organization runs its own custom data workflow and needs help consider these or other workflow approaches, contact us to discuss Interline’s <a href="https://www.interline.io/consulting/">consulting service</a>. We’re glad to share our experiences with Argo, Kubernetes, and related technologies.</li></ul>]]></content:encoded></item></channel></rss>