<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0"><channel><title><![CDATA[Drew Dara-Abrams — Interline Technologies]]></title><description><![CDATA[We're a product and consulting business that helps organizations understand and improve transportation networks, digitally.]]></description><link>https://www.interline.io/</link><image><url>https://www.interline.io/favicon.ico</url><title>Drew Dara-Abrams — Interline Technologies</title><link>https://www.interline.io/</link></image><lastBuildDate>Fri, 28 Apr 2023 07:00:00 GMT</lastBuildDate><atom:link href="https://www.interline.io/blog/author/drew/rss.xml" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[US Federal Transit Administration finalizes plan to collect GTFS URLs in US National Transit Database]]></title><description><![CDATA[To meet meet the Bipartisan Infrastructure Law's requirement to collect data on American transit agencies' "geographic service area coverage," the US Federal Transit Administration will begin to require GTFS feeds from fixed-route transit agencies that receive federal funds.]]></description><link>https://www.interline.io/blog/us-ntd-reporting-gtfs-adopted/</link><guid isPermaLink="false">67255784b62841000136edf4</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 28 Apr 2023 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/bil-geographic-service-coverage-area.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/bil-geographic-service-coverage-area.png" alt="US Federal Transit Administration finalizes plan to collect GTFS URLs in US National Transit Database"><p>The US Bipartisan Infrastructure Law passed by Congress and signed into law by President Biden in 2021 was big, very big. While most people have focused on the large sums of funding that are now becoming available to improve American transportation infrastructure, we’ve focused on one of the details: The “BIL” requires the US Federal Transit Administration to collect data on American transit agencies’ “geographic service area coverage.”</p><figure><img src="https://www.interline.io/content/images/2024/11/bil-geographic-service-coverage-area-1.png" alt="first page of FTA's notice and request for comments regarding additions to the NTD reporting process" width="1130" height="1462"><figcaption><p>The US Bipartian Infrastructure Law, formally known as the <a href="https://www.congress.gov/bill/117th-congress/house-bill/3684/text">Infrastructure Investment and Jobs Act</a>, added a requirement that the US National Transit Database collect data on &quot;geographic service area coverage&quot; for American transit agencies receiving federal funds.</p></figcaption></figure><h2 id="proposal">Proposal</h2><p>To meet this requirement, the FTA issued a proposal on July 7, 2022 to collect static GTFS feeds from all American transit agencies that receive federal funds and operate fixed-route transit service. The <a href="https://www.interline.io/blog/us-ntd-reporting-gtfs/">Interline team submitted feedback to FTA</a> based on our experience operating the Transitland open transit data platform, which aggregates GTFS feeds from nearly a thousand transit operators in the US (and over 3,000 when including other countries).</p><h2 id="adoption">Adoption</h2><p>On March 3, 2023, the FTA released responses to feedback and formally adopted its proposal, with no changes made to the GTFS reporting requirement. For more information see <a href="https://www.federalregister.gov/documents/2023/03/03/2023-04379/national-transit-database-reporting-changes-and-clarifications">this notice in the US Federal Register</a>.</p><h2 id="timing">Timing</h2><p>While the NTD reporting requirements have now been formally changed, agencies will have some time until they begin submitting their GTFS feeds. In response to a question from Interline about timing, the NTD team tells us that:</p><blockquote><p>NTD will begin collecting GTFS data in Report Year 2023. Agencies will submit 2023 data to the NTD beginning in October of 2023, and the associated RY23 data products containing GTFS data will be released in Fall of 2024.</p></blockquote><h2 id="data-release">Data release</h2><p>When US National Transit Database Report Year 2023 data products are released, they will be available on the <a href="https://www.transit.dot.gov/ntd/ntd-data">NTD Data</a> website.</p><h2 id="improvements-big-and-small">Improvements big and small</h2><p>The word that’s been most often said in recent years about the BIL is that it’s a “generational” amount of funding and policy change for American physical infrastructure and transportation networks.</p><p>The collection and dissemination of URLs for GTFS feeds by the US Federal Transit Administration may not be as massive of a change in dollar amounts. Yet, this change still reflects a substantive development in the way that the US federal government engages with GTFS. We’re looking forward to seeing how FTA and other parts of the US Department of Transportation continue to promote and engage with transit data.</p>]]></content:encoded></item><item><title><![CDATA[Updated GTFS-Fares v2 for the SF Bay Area]]></title><description><![CDATA[MTC and Interline have released updated fares and transfer discounts for over 30 Bay Area transit agencies using the adopted GTFS-Fares v2 "base implementation" and a series of proposed extensions. ]]></description><link>https://www.interline.io/blog/mtc-regional-gtfs-feed-fares-updates/</link><guid isPermaLink="false">672550a2b62841000136ed3b</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Tue, 04 Oct 2022 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/ios-directions-with-fares-2.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/ios-directions-with-fares-2.png" alt="Updated GTFS-Fares v2 for the SF Bay Area"><blockquote><p><strong>April 20, 2023</strong>: The GTFS-Fares v2 extension originally named <code>fare_containers.txt</code> has been renamed to <code>fare_media.txt</code>, as part its recent <a href="https://github.com/google/transit/pull/355">adoption in the GTFS specification</a>.</p></blockquote><p>In 2020, Interline and the Metropolitan Transportation Commission <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-additions/#fares-and-transfer-discounts">released</a> a beta version of fares and transfer discounts for eight SF Bay Area transit agencies. Since then, we’ve been working together to improve data coverage and its schema. As of this August, the Regional GTFS Feed now includes fares and transfer discounts for over 30 Bay Area transit agencies. The updated dataset uses the adopted GTFS-Fares v2 “base implementation,” as well a few proposed additions to the GTFS-Fares v2 data schema. This data is now available for use, and our team is updating it on an ongoing basis.</p><p>In this blog post we’ll cover:</p><ul><li><a href="#gtfs-fares-v2-in-action">GTFS-Fares v2 in action</a></li><li><a href="#gtfs-fares-v2-base-implementation-and-future-additions-to-the-specification">GTFS-Fares v2 base implementation and future additions to the specification</a></li><li><a href="#sf-bay-area-transit-agencies-with-gtfs-fares-v2-data">SF Bay Area transit agencies with GTFS-Fares v2 data</a></li><li><a href="#using-gtfs-fares-v2-data-for-the-bay-area">Using GTFS-Fares v2 data for the Bay Area</a></li></ul><h2 id="gtfs-fares-v2-in-action">GTFS-Fares v2 in action</h2><p>If you have an Apple iPhone running <a href="https://www.apple.com/ios/ios-16/">iOS 16</a>, you can see the estimated costs of transit journeys. This data is sourced from the the Regional Feed.</p><p>All of the above fares information is defined in the Regional GTFS Feed using the GTFS-Fares v2 specification. Here are extracts of the relevant GTFS files from the feeds for the itinerary displayed on the iPhone:</p><h3 id="fare_productstxt">fare_products.txt</h3><table><thead><tr><th>fare_product_id</th><th>fare_product_name</th><th>amount</th><th>currency</th><th>rider_category_id</th><th>fare_media_id</th></tr></thead><tbody><tr><td><code>AC:local:single</code></td><td>AC Transit local fare</td><td>2.25</td><td>USD</td><td><code>adult</code></td><td><code>clipper</code></td></tr><tr><td><code>BA:matrix:FTVL-EMBR</code></td><td>generated</td><td>2.15</td><td>USD</td><td><code>adult</code></td><td><code>clipper</code></td></tr></tbody></table><h3 id="fare_mediatxt-updated">fare_media.txt Updated</h3><table><thead><tr><th>fare_media_id</th><th>fare_media_name</th><th>fare_media_type</th></tr></thead><tbody><tr><td><code>clipper</code></td><td>Clipper</td><td>2</td></tr></tbody></table><h3 id="fare_leg_rulestxt">fare_leg_rules.txt</h3><table><thead><tr><th>leg_group_id</th><th>from_area_id</th><th>to_area_id</th><th>network_id</th><th>fare_product_id</th></tr></thead><tbody><tr><td><code>AC</code></td><td><code>AC:local</code></td><td><code>AC:local</code></td><td><code>AC</code></td><td><code>AC:local:single</code></td></tr><tr><td><code>BA</code></td><td><code>FTVL</code></td><td><code>EMBR</code></td><td><code>BA</code></td><td><code>BA:matrix:FTVL-EMBR</code></td></tr></tbody></table><h3 id="stop_areastxt">stop_areas.txt</h3><table><thead><tr><th>area_id</th><th>stop_id</th></tr></thead><tbody><tr><td><code>AC:local</code></td><td><code>55227</code></td></tr><tr><td><code>AC:local</code></td><td><code>55550</code></td></tr><tr><td><code>FTVL</code></td><td><code>FTVL</code></td></tr><tr><td><code>EMBR</code></td><td><code>EMBR</code></td></tr></tbody></table><p>Note that this data schema has changed since the examples in <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-additions/#an-example-of-fares-v2-in-practice">our first release blog post</a>. This new schema was adopted as part of the GTFS-Fares v2 “base implementation,” which we will now discuss in more detail.</p><h2 id="gtfs-fares-v2-base-implementation-and-future-additions-to-the-specification">GTFS-Fares v2 base implementation and future additions to the specification</h2><p>Since the <a href="https://bit.ly/gtfs-fares">original GTFS-Fares v2 proposal</a>, many stakeholders in GTFS have provided additional input. In May, stakeholders voted to adopt the core of the refined GTFS-Fares v2 proposal as a <a href="https://github.com/google/transit/pull/286">“base implementation”</a>.</p><p>The “base implementation” simplifies the organization of <code>fare_leg_rules.txt</code>, <code>fare_transfer_rules.txt</code>, and <code>fare_products.txt</code>. This is a breaking change, so consumers who implemented their GTFS processing software against the original proposal will need to update their tools.</p><p>To fully capture the complexity of Bay Area transit fares and transfer discounts requires additional data tables and fields that were included in the original proposal but not in the “base implementation.”</p><p>Here is an overview of all the GTFS-Fares v2 tables and columns/fields included in the Regional GTFS Feed:</p><table><thead><tr><th>GTFS file</th><th>Spec version(s)</th><th>Columns added for extensions</th><th>Documentation</th></tr></thead><tbody><tr><td><code>fare_products.txt</code></td><td>&quot;base&quot; with additional columns for extensions</td><td><ul><li><code>rider_category_id</code></li></ul></td><td><a href="https://gtfs.org/schedule/reference/#fare_productstxt">See gtfs.org reference</a></td></tr><tr><td><code>fare_leg_rules.txt</code></td><td>&quot;base&quot; with additional columns for extensions</td><td><ul><li><code>transfer_only</code>*</li></ul></td><td><a href="https://gtfs.org/schedule/reference/#fare_leg_rulestxt">See gtfs.org reference</a></td></tr><tr><td><code>fare_transfer_rules.txt</code></td><td>&quot;base&quot; with additional columns for extensions</td><td><ul><li><code>filter_fare_product_id</code>*</li></ul></td><td><a href="https://gtfs.org/schedule/reference/#fare_transfer_rulestxt">See gtfs.org reference</a></td></tr><tr><td><code>areas.txt</code></td><td>&quot;base&quot;</td><td></td><td><a href="https://gtfs.org/schedule/reference/#areastxt">See gtfs.org reference</a></td></tr><tr><td><code>stop_areas.txt</code></td><td>&quot;base&quot;</td><td></td><td><a href="https://gtfs.org/schedule/reference/#stop_areastxt">See gtfs.org reference</a></td></tr><tr></tr><tr><td><code>fare_media.txt</code></td><td>&quot;base&quot; as of March 14, 2023 Updated</td><td></td><td><a href="https://gtfs.org/schedule/reference/#fare_mediatxt">See gtfs.org reference</a></td></tr><tr><td><code>rider_categories.txt</code></td><td>extension</td><td><ul><li><code>rider_category_id</code></li><li><code>rider_category_name</code></li><li><code>min_age</code></li><li><code>max_age</code></li><li><code>eligibility_url</code></li></ul></td><td><a href="https://docs.google.com/document/d/19j-f-wZ5C_kYXmkLBye1g42U-kvfSVgYLkkG5oyBauY/edit#heading=h.9gqyl4w7f6x1">See proposal Google Doc</a></td></tr><tr><td><code>routes.txt</code></td><td>extension</td><td><ul><li><code>as_route</code></li></ul></td><td><a href="https://docs.google.com/document/d/19j-f-wZ5C_kYXmkLBye1g42U-kvfSVgYLkkG5oyBauY/edit#heading=h.v160xvt5oen5">See proposal Google Doc</a></td></tr></tbody></table><p>* <code>fare_leg_rules.txt transfer_only</code> and <code>fare_transfer_rules.txt filter_fare_product_id</code> are two fields proposed for describing partial credit on inter-agency transfers. Thanks to the <a href="https://transitapp.com/">Transit</a> app team for proposing these additions to make some of the transfer discounts in the Regional GTFS Feed more explicit for consuming software to parse.</p><p>Interline and MTC will continue to produce these “extensions” to the “base” data schema, and we’ll continue to work with data consumers, <a href="https://mobilitydata.org/">MobilityData</a>, and other stakeholders to finalize their adoption into the GTFS spec.</p><h2 id="sf-bay-area-transit-agencies-with-gtfs-fares-v2-data">SF Bay Area transit agencies with GTFS-Fares v2 data</h2><p>Interline and MTC now produce fares, intra-agency transfer discounts, and inter-agency transfer discounts for the following agencies:</p><table><thead><tr><th>511 operator code</th><th>Operator full name</th></tr></thead><tbody><tr><td><code>3D</code></td><td>Tri Delta Transit</td></tr><tr><td><code>AC</code></td><td>AC Transit (including transbay routes)</td></tr><tr><td><code>AF</code></td><td>Angel Island Tiburon Ferry</td></tr><tr><td><code>AM</code></td><td>Capitol Corridor</td></tr><tr><td><code>BA</code></td><td>Bay Area Rapid Transit (BART)</td></tr><tr><td><code>CC</code></td><td>County Connection</td></tr><tr><td><code>CE</code></td><td>Altamont Corridor Express (ACE)</td></tr><tr><td><code>CM</code></td><td>Commute.org Shuttles</td></tr><tr><td><code>CT</code></td><td>Caltrain</td></tr><tr><td><code>DE</code></td><td>Dumbarton Express Consortium</td></tr><tr><td><code>EM</code></td><td>Emery Go-Round</td></tr><tr><td><code>FS</code></td><td>Fairfield and Suisun Transit</td></tr><tr><td><code>GF</code></td><td>Golden Gate Ferry</td></tr><tr><td><code>GG</code></td><td>Golden Gate Transit</td></tr><tr><td><code>MA</code></td><td>Marin Transit</td></tr><tr><td><code>MV</code></td><td>MVgo Mountain View</td></tr><tr><td><code>PE</code></td><td>Petaluma Transit</td></tr><tr><td><code>RV</code></td><td>Rio Vista Delta Breeze</td></tr><tr><td><code>SA</code></td><td>Sonoma Marin Area Rail Transit</td></tr><tr><td><code>SB</code></td><td>San Francisco Bay Ferry</td></tr><tr><td><code>SC</code></td><td>Valley Transportation Authority (VTA)</td></tr><tr><td><code>SF</code></td><td>San Francisco Municipal Transportation Agency (SFMTA)</td></tr><tr><td><code>SI</code></td><td>San Francisco International Airport (SFO)</td></tr><tr><td><code>SM</code></td><td>SamTrans</td></tr><tr><td><code>SO</code></td><td>Sonoma County Transit</td></tr><tr><td><code>SR</code></td><td>Santa Rosa CityBus</td></tr><tr><td><code>SS</code></td><td>City of South San Francisco</td></tr><tr><td><code>ST</code></td><td>SolTrans</td></tr><tr><td><code>TD</code></td><td>Tideline Water Taxi</td></tr><tr><td><code>UC</code></td><td>Union City Transit</td></tr><tr><td><code>VC</code></td><td>Vacaville City Coach</td></tr><tr><td><code>VN</code></td><td>VINE Transit</td></tr><tr><td><code>WC</code></td><td>Western Contra Costa (WestCat)</td></tr><tr><td><code>WH</code></td><td>Livermore Amador Valley Transit Authority (LAVTA)</td></tr></tbody></table><p>Fares are included for Clipper Card, cash, and a wide variety of passes. (<a href="https://www.clippercard.com/">Clipper Card</a> is the contact-less payment card and system operated by MTC and available on buses, trains, and ferries throughout the Bay Area.) Transfer discounts between routes operated by the same agency (intra-agency transfers) are included for all methods of payment. Transfer discounts between routes operated by different agencies (inter-agency transfers) are only included for Clipper Card.</p><p>Our team is updating these fares and transfer discounts on a monthly basis.</p><h2 id="using-gtfs-fares-v2-data-for-the-bay-area">Using GTFS-Fares v2 data for the Bay Area</h2><p>Download the daily Regional Feed like so:</p><ol><li><a href="https://511.org/open-data/token">Sign up for a 511 Open Data API token</a></li><li>Download from <code>http://api.511.org/transit/datafeeds?api_key=[your_key]&amp;operator_id=RG</code></li></ol><p>We welcome questions sent to the <a href="https://groups.google.com/forum/#!forum/511sfbaydeveloperresources">511SFBayDeveloperResources mailing list</a>.</p><h2 id="acknowledgements">Acknowledgements</h2><p>Credit and many thanks to project team members including Ian Rees (Interline), Nome Dickerson (Garnet Consulting), Nisar Kapeel and Kapeel Daryani (MTC), and our partners at Bay Area transit agencies.</p>]]></content:encoded></item><item><title><![CDATA[US National Transit Database to collect GTFS URLs]]></title><description><![CDATA[American public transit agencies that receive federal funds are required to report information for inclusion in the National Transit Database. The Federal Transit Administration is proposing that each agency will also provide a URL for a GTFS feed in their NTD reports.]]></description><link>https://www.interline.io/blog/us-ntd-reporting-gtfs/</link><guid isPermaLink="false">672557c9b62841000136ee02</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Wed, 07 Sep 2022 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/us-ntd-federal-register.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/us-ntd-federal-register.png" alt="US National Transit Database to collect GTFS URLs"><p><strong>April 28, 2023</strong>: The Federal Transit Administration has now adopted these new reporting requirements. For a brief overview, see our more recent <a href="https://www.interline.io/blog/us-ntd-reporting-gtfs-adopted/">blog post describing when FTA will begin requiring US transit agencies to submit URLs for their GTFS feeds to the National Transit Database</a>.</p><p>American public transit agencies that receive federal funds are required to report information for inclusion in the <a href="https://www.transit.dot.gov/ntd">National Transit Database</a>. The Federal Transit Administration, which administers to NTD, is <a href="https://www.regulations.gov/document/FTA-2022-0018-0001">proposing</a> that each agency will also provide a URL for a GTFS feed in their NTD reports. Submitted GTFS feeds will enter the public domain. This is great news for all producers and consumers of GTFS data.</p><figure><img src="https://www.interline.io/content/images/2024/11/us-ntd-federal-register-1.png" alt="first page of FTA's notice and request for comments regarding additions to the NTD reporting process" width="1166" height="1516"><figcaption><p><a href="https://www.regulations.gov/document/FTA-2022-0018-0001">Notice and request for comments</a> on NTD reporting requirements from the Federal Transit Administration.</p></figcaption></figure><p>We’ve submitted feedback to the FTA based on our experience building and operating Transitland, and we’re sharing it here as well:</p><blockquote><p>At Interline Technologies, we specialize in software services for GTFS, GTFS Realtime, and related data specifications. Our Transitland data platform (<a href="https://www.transit.land/">www.transit.land</a>) aggregates open-data feeds from over 3,000 operators in over 50 countries, including 908 operators in the United States. We serve organizations that both produce and consume these data feeds. Our clients include public transit agencies, private mobility providers, planning firms, and academic research institutions. This is just a sampling of the many types of firms that make use of GTFS and related data feeds.</p></blockquote><blockquote><p>Based on our own experience, as well as the experience of our clients, we see much value in FTA’s proposed addition to the NTD reporting process to have agencies provide a public GTFS feed. Increasing the number and accessibility of openly licensed GTFS feeds with stable URLs from American transit agencies will have many benefits. In addition to being useful for transit trip planning (the best known use-case for GTFS), this will also assist with transportation planning, land-use planning, real-estate assessment, academic research, and local advocacy. Coverage of the largest and the smallest transit agencies is important for many of these endeavors, as is comprehensive coverage of both urban and rural operators. Using NTD reporting to promote the production and open sharing of GTFS data feeds will assist with all of these use-cases.</p></blockquote><blockquote><p>We are pleased to see that agencies will be encouraged to report a stable URL that is updated on their own servers. Transitland and similar data aggregators operate best when the “source of truth” for each feed is maintained on each agency’s servers and the aggregator fetches from that location on a regular basis.</p></blockquote><blockquote><p>We encourage FTA to consider multiple means of distributing the lists of GTFS feed URLs. While most NTD data is included in a single Excel file with multiple sheets, we would propose that FTA also disseminate a separate file that just lists GTFS feed URLs. Ideally this would be in the CSV format, be hosted at a stable URL, and include the associated NTD IDs for each GTFS feed URL. In addition to the NTD ID, we suggest also listing the agency’s ID used on the NTD website for agency profile pages. For example, the final portion of <code>https://www.transit.dot.gov/ntd/transit-agency-profiles/san-francisco-bay-area-rapid-transit-district</code></p></blockquote><blockquote><p>FTA may consider posting a CSV listing the feed URLs, associated NTD IDs, and NTD website IDs to an open-data portal such as data.gov on an ongoing basis. This will be useful for a very wide range of users.</p></blockquote><blockquote><p>At present, our staff and external contributors maintain associations between GTFS feed URLs and NTD IDs in the Transitland Atlas repository on GitHub. It is open to public use and contributions at <a href="https://github.com/transitland/transitland-atlas">https://github.com/transitland/transitland-atlas</a> While we believe this structure is likely more complex than is necessary for FTA and NTD’s purposes, we offer this as one example for how to provide information on many feeds, along with metadata about each operator and (when relevant) its US NTD ID.</p></blockquote><figure><img src="https://www.interline.io/content/images/2024/11/bart-operator-page-screenshot.png" alt="screenshot of BART operator detail page listing its US NTD ID" width="2000" height="1228"><figcaption><p>US NTD ID listed on <a href="https://www.transit.land/operators/o-9q9-bart">Transitland's operator detail page for BART</a>.</p></figcaption></figure><blockquote><p>Finally, we strongly support the proposed change that “upon publication, all GTFS data submitted to the NTD will enter the public domain.” Since 2014, our team has been working to improve the state of licensing for GTFS data. At our last company (Mapzen) we prepared a model license for GTFS feeds and discussed it in depth with USDOT OST staff. At Interline, we continue to assess the state of licenses and terms attached to public transit data. In the Transitland Atlas repository mentioned previously, our staff maintain metadata about the licenses attached to many of the feeds. This is important for our institutional clients, so they can be assured that we and they are making a good faith effort to follow the license and terms for each agency. However, all of this effort to track and review such a wide range of licenses is resource intensive. We do not think it is a good use of anyone’s time. Furthermore, we do not believe it benefits or protects transit agencies that produce GTFS data. GTFS feeds contain no privileged data, nor do they contain personally identifiable information (PII). Publishing a GTFS feed in the public domain does not entail giving up other protections such as a transit agency’s trademark of their brand name and logo. Transit agencies benefit when their GTFS feeds are useful widely by as many consumers as possible. We are pleased to see FTA proposing to simplify this overly complicated situation by instead requiring that GTFS feeds submitted to the NTD reporting process enter the public domain.</p></blockquote><blockquote><p>Thank you for taking these comments. You are welcome to reach us at <a href="mailto:info@interline.io">info@interline.io</a> if we can provide any more information.</p></blockquote>]]></content:encoded></item><item><title><![CDATA[Observed stop arrivals and departures through the SF Bay Area]]></title><description><![CDATA[Interline and the Metropolitan Transportation Commission are now releasing monthly "stop observation" datasets derived from the Regional GTFS Realtime Feed.]]></description><link>https://www.interline.io/blog/mtc-regional-stop-observations/</link><guid isPermaLink="false">672554e4b62841000136ed97</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 15 Jul 2022 22:25:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/trip-update-selection.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/trip-update-selection.png" alt="Observed stop arrivals and departures through the SF Bay Area"><p>When did the bus arrived at that stop yesterday? This isn’t a question that transit riders ask often — but it is a question that analysts and planners often do ask.</p><p>Interline and the Metropolitan Transportation Commission have been archiving and processing the <a href="https://www.interline.io/blog/mtc-regional-gtfs-realtime-feed/">Regional GTFS Realtime</a> feed in order to answer these questions.</p><h2 id="what-are-stop-observations">What are stop observations?</h2><p>“Stop observations” describes transit service as was delivered to riders: the arrival time of a vehicle at a designated stop, along with any additional contextual information available from GTFS and GTFS Realtime feeds.</p><p>Stop observations are in contrast with the <code>stop_times.txt</code> file in the <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-release/">daily Regional Feed</a>, which uses static GTFS data to capture transit service as it is planned and scheduled. This is also in contrast with the <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-additions/#historical-feeds">monthly Historic Regional Feed</a>, which uses static GTFS data to capture transit service schedules, retrospectively, as they changed over an entire month (with a different schedule for each agency for each day).</p><p>Stop observations can be used to better understand the real-world realities of transit service as it fluctuates from day to day and location to location. Stop observations can be used to estimate on-time performance (also known as schedule adherence or punctuality) and to inform potential schedule adjustments by service planners. Stop observations can also be used by navigation app developers to provide their users with contextual information, such as an estimated range of times when a route is likely to depart from a given stop.</p><h2 id="measurement-error-and-imprecision">Measurement error and (im)precision</h2><p>Stop observations are inferred from public GTFS Realtime data feeds (trip updates and vehicle positions) and therefore, there are many potential sources for measurement error. Potential sources of error include GPS error on transit vehicles, misassignment of vehicles to “runs” of a given trip and route, imprecision or data errors in the transit agency’s computer-aided dispatch/automatic vehicle location (CAD/AVL) system, issues in the transit agency’s GTFS Realtime emitting system, issues in 511.org’s GTFS Realtime aggregation system, or issues in the <a href="https://www.interline.io/blog/mtc-regional-gtfs-realtime-feed/">Regional Realtime GTFS Feed</a>.</p><p>We’ll discuss how stop observations are calculated later in this blog post, to help users better understand stop observations as measurements. When considering potential applications for stop observations, we ask users to keep the limitations of this data source in mind.</p><h2 id="what-information-is-contained-in-a-stop-observation">What information is contained in a stop observation?</h2><p>We are distributing stop observations in the Historic Regional Feed. When users request a Historic Regional Feed with stop observations for a given year and month, a <code>stop_observations.txt</code> file will be included with all stop observations for that month.</p><p>The <code>stop_observations.txt</code> file is a CSV file based on the <a href="https://bit.ly/gtfs-performance">GTFS-Performance draft proposal</a>. GTFS-Performance is an extension to the GTFS specification prepared by Remix, Swiftly, and their partners.</p><p>The following table is based on the GTFS-Performance draft proposal and covers columns included in our implementation of <code>stops_observations.txt</code>, as well as columns that are in the proposal but not in our implementation:</p><table><thead><tr><th>Field Name</th><th>Details</th><th>Regional GTFS stop observations support</th></tr></thead><tbody><tr><td><code>trip_id</code></td><td><p>Identifies a trip.</p><p> When <code>schedule_relationship=DUPLICATED</code>, then this references the trip that is copied, like GTFS Realtime <code>TripUpdate.trip.trip_id</code>. </p></td><td><p>✅</p></td></tr><tr><td><code>trip_start_time</code></td><td><p> See GTFS Realtime <code>TripDescriptor.start_time</code>. </p><p>Required to specify the start time of a trip that’s not defined in Static GTFS which occurs when using frequency-based service defined by <code>frequencies.txt</code> or when an ad-hoc added service based on an existing trip via <code>TripUpdate.tripschedule_relationship=DUPLICATED</code>.</p></td><td><p>✅ (when provided in agency source RT feed)</p></td></tr><tr><td><code>stop_id</code></td><td><p>Identifies a stop</p></td><td>✅ (added by Interline for developer convenience)</td></tr><tr><td><code>stop_sequence</code></td><td><p>Identifies the order of stops for a particular trip.</p></td><td><p>✅</p></td></tr><tr><td><code>schedule_relationship</code></td><td><p>Valid options are:</p><ul><li> 0 - SCHEDULED </li><li> 1 - SKIPPED </li><li> 2 - NO_DATA </li><li> 3 - UNSCHEDULED </li><li> 4 - CANCELED </li><li> 5 - DUPLICATED </li><li>6 - MODIFIED</li></ul></td><td><p>✅</p></td></tr><tr><td><code>schedule_adherence_stop</code></td><td><p>Indicates if this observation is used for reporting purposes when a subset of stops are utilized, instead of all stops.</p></td><td><p>No</p></td></tr><tr><td><code>trip_start_date</code></td><td><p>The start date of this trip instance in <code>YYYYMMDD</code> format to specify the service date of the corresponding trip. This should match the value in <code>TripDescriptor.start_date</code>.</p></td><td><p>✅</p></td></tr><tr><td><code>vehicle_id</code></td><td><p> Unique identifier for the vehicle that served this stop. This should match the value provided in real-time via <code>VehicleDescriptor.id</code>. </p></td><td><p>✅ (when provided in agency source RT feed)</p></td></tr><tr><td><code>scheduled_arrival_time</code></td><td><p> GTFS static scheduled arrival time </p></td><td><p>✅ (added by Interline for developer convenience)</p></td></tr><tr><td><code>observed_arrival_time</code></td><td><p>Observed arrival time at a specific stop for a specific trip on a route. For times occurring after midnight on the service day, enter the time as a value greater than <code>24:00:00</code> in <code>HH:MM:SS</code> local time for the day on which the trip schedule begins.</p></td><td><p>✅</p></td></tr><tr><td><code>observed_departure_time</code></td><td><p>Observed departure time at a specific stop for a specific trip on a route. For times occurring after midnight on the service day, enter the time as a value greater than <code>24:00:00</code> in <code>HH:MM:SS</code> local time for the day on which the trip schedule begins.</p></td><td><p>✅ (when available)</p></td></tr><tr><td><code>new_departure_time</code></td><td><p>Required when <code>schedule_relationship=DEPARTURE_TIME_MODIFIED</code>.</p></td><td><p>No</p></td></tr><tr><td><code>dwell_time_secs</code></td><td><p>Seconds spend boarding/alighting riders at the stop.</p></td><td><p>No</p></td></tr><tr><td><code>scheduled_dwell_time_secs</code></td><td><p>Seconds scheduled for boarding/alighting riders at the stop (when provided in agency source RT feed).</p></td><td><p>✅</p></td></tr><tr><td><code>boardings</code></td><td><p>See <a href="https://gtfsride.org/">GTFS-ride</a>'s board_alight.boardings</p></td><td><p>No</p></td></tr><tr><td><code>alightings</code></td><td><p>See <a href="https://gtfsride.org/">GTFS-ride</a>'s board_alight.alightings</p></td><td><p>No</p></td></tr><tr><td><code>source</code></td><td><p>Valid options are:</p><ul><li>0 - Manual</li><li>1 - Door Sensor</li><li>2 - Estimated using GPS/AVL data</li></ul></td><td><p>No</p></td></tr><tr><td><code>uncertainty</code></td><td><p>See <code>StopTimeEvent.uncertainty</code>.</p></td><td><p>✅</p></td></tr><tr><td><code>occupancy_status</code></td><td><p>See <code>VehiclePosition.occupancy_status</code>.</p></td><td><p>✅ (when provided in agency source RT feed)</p></td></tr><tr><td><code>occupancy_percentage</code></td><td><p>See <code>VehiclePosition.occupancy_percentage</code>.</p></td><td><p>✅ (when provided in agency source RT feed)</p></td></tr><tr><td><code>occupancy_type</code></td><td><p>See <a href="https://gtfsride.org/">GTFS-ride</a>'s <code>board_alight.load_type</code>.</p></td><td><p>No</p></td></tr><tr><td><code>occupancy_source</code></td><td><p>See <a href="https://gtfsride.org/">GTFS-ride</a>'s <code>board_alight.source</code>.</p></td><td><p>No</p></td></tr></tbody></table><h2 id="how-are-stop-observations-calculated">How are stop observations calculated?</h2><p>Stop observations are derived retrospectively using an archive of the Regional GTFS Realtime Feed (updated every 15 seconds). Once a day, our workflow processes GTFS Realtime trip updates and vehicle positions from the archive, combined together with the day’s static GTFS daily regional feed.</p><p><strong>Deriving stop observations from trip updates:</strong> When an agency supplies GTFS Realtime trip updates, our workflow identifies the final <code>TripUpdate</code> sent before a vehicle reaches a given stop. From that record, our workflow extracts the most recent <code>StopTimeUpdate</code>. Here is an example showing three trip updates fetched from BART. The first two trip updates are before the train reaches Montgomery Station; the third trip update is after the train departs Montgomery Station. The portion of all of the records that is use to create a stop observation is marked with the orange box:</p><figure><img src="https://www.interline.io/content/images/2024/11/trip-update-selection-1.png" alt="illustration of three GTFS Realtime feed trip updates, showing the portion of stop time updates retained to create a stop observation record" width="1167" height="943"></figure><p><strong>Deriving stop observations from vehicle positions</strong>: When an agency supplies vehicle positions for a given route, our workflow also identifies the last update sent before the stop location. To do this, we need to consider the geometry of the route alignment from <code>shapes.txt</code> in the static RG feed. The workflow snaps the latitude/longitude point from each vehicle position to the route alignment, enabling it to determine from which update to take the <code>StopTimeUpdate</code> portion. Like many geometric algorithms, this approach can be stymied by complicated paths, such as looping routes that visit the same station multiple times. Our workflow can take advantage of stop sequence, when specified. Still, the algorithm can only do its best to estimate when the route alignment is complicated and/or the update frequency is infrequent.</p><p>By being able to derive stop observations from two different parts of GTFS Realtime feeds, we have also been able to test and tune our workflow, to ensure that the two approaches agree when source feeds are of high quality.</p><h2 id="related-data-specifications">Related data specifications</h2><p>Our stop observations data model is based on the <a href="http://bit.ly/gtfs-performance">GTFS-Performance</a> proposal. The GTFS-Performance proposal is, in turn, inspired by the larger <a href="https://apps.trb.org/cmsfeed/TRBNetProjectDisplay.asp?ProjectID=4687">TCRP G-18</a> project. The GTFS-Performance <code>stop_visits.txt</code> is supposed to represent a subset of the TCRP G-18 <code>stop_visits.txt</code> file. Thank you to the creators of GTFS-Performance and to the G-18 research team.</p><p>GTFS-Performance and TCRP G-18 go well beyond the functionality that we have implemented in our system for producing stop observations, to also address more internal operating concerns of transit agencies. We have intentionally implemented our system to create stop observations using only publicly available GTFS and GTFS Realtime inputs.</p><h2 id="accessing-stop-observations">Accessing stop observations</h2><p>Stop observations are now included in historical monthly RGs. To download a historical monthly RG including stop observations, make an HTTP request like so:</p><pre><code>wget https://api.511.org/transit/datafeeds?operator_id=RG&amp;historic=2022-03-so&amp;api_key=xxx
</code></pre><p>Replace <code>2022-03</code> with a given year and month. Note the addition of <code>-so</code> to request the inclusion of stop observations. Note that <code>wget</code> is a Unix command that can be replaced by any other utility to download the URL; also note that <code>xxx</code> should be replaced by your own <a href="https://511.org/open-data/token">511.org Open Data API key</a>.</p><p>Monthly RGs without stop observations are roughly 50Mb in size; with stop observations, the monthly RG will be roughly 500Mb to download as a compressed ZIP archive and then roughly 2Gb of uncompressed CSV files.</p><p>Stop observations are available on a more frequent basis to staff at MTC and Bay Area transit agencies. Transit agencies may contact the 511 SF Bay Data Portal at <a href="mailto:transitdata@511.org">transitdata@511.org</a> for more information.</p><h2 id="using-stop-observations-to-understand-transit">Using stop observations to understand transit</h2><p>Public-transit agencies can be hesitant to release this type of data. In part, it’s a challenge to assemble operations data in an ongoing and consistent manner. In part, it’s a risk that the general public will take operations data out of context.</p><p>While we began this blog post by asking <em>When did the bus arrived at that stop yesterday?</em>, we hope that users of stop observations will ask an even wider range of questions of this dataset, especially questions that can explore the complexity of daily operations for over 20 agencies operating buses, trains, subways, and ferries across the Bay Area and questions that can help to improve the experience for riders using navigation apps and similar software.</p>]]></content:encoded></item><item><title><![CDATA[SF Bay Area's Regional GTFS Realtime Feed]]></title><description><![CDATA[The Metropolitan Transportation Commission and Interline are pleased to announce a single consolidated Regional GTFS Realtime feed. Developers can now get a comprehensive and up-to-date views of the Bay Area's bus, train, and ferry services.]]></description><link>https://www.interline.io/blog/mtc-regional-gtfs-realtime-feed/</link><guid isPermaLink="false">6725541ab62841000136ed7f</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 10 Sep 2021 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/list-of-rt-enabled-agencies.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/list-of-rt-enabled-agencies.png" alt="SF Bay Area's Regional GTFS Realtime Feed"><p>Last year, <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-release/">Interline and the Metropolitan Transportation Commission released the Regional GTFS Feed for the San Francisco Bay Area</a> and also a <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-additions/">series of improvements to the static data including station pathways and fares</a>. We’re now pleased to announce a single consolidated Regional GTFS Realtime feed.</p><p>Developers can combine the static Regional GTFS Feed and the Regional GTFS Realtime endpoints for comprehensive and up-to-date views of the Bay Area’s bus, train, and ferry services. Combining feeds is straightforward, as the same ID schemes are used for data entities across all feeds.</p><h2 id="_21-agencies-included">21 agencies included</h2><p>The Regional GTFS Realtime Feed currently aggregates data for the following Bay Area transit agencies:</p><table><thead><tr><th>Operator Name</th><th>Service Alerts</th><th>Trip Updates</th><th>Vehicle Positions</th></tr></thead><tbody><tr><td>AC Transit</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Bay Area Rapid Transit (BART)</td><td>✅</td><td>✅</td><td>-</td></tr><tr><td>Caltrain</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>County Connection</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Dumbarton Express Consortium</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Emery Go-Round</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Fairfield and Suisun Transit</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Golden Gate Transit (GGT)</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Livermore Amador Valley Transit Authority (LAVA)</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Marin Transit</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Petaluma Transit</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>SamTrans</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>San Francisco Municipal Transportation Agency (SFMTA)</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Santa Rosa CityBus</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>SolTrans</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Sonoma County Transit</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Sonoma Marin Area Rail Transit (SMART)</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Tri Delta Transit</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Union City Transit</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>VINE Transit</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>Valley Transportation Authority (VTA)</td><td>✅</td><td>✅</td><td>✅</td></tr></tbody></table><p>MTC will be announcing additional agencies providing GTFS Realtime data in the future.</p><h2 id="built-using-shared-tooling-and-validation-processes">Built using shared tooling and validation processes</h2><p>Like the static Regional GTFS Feed, the Regional GTFS Realtime Feed is build using Interline’s open-source tooling, including the <a href="https://github.com/interline-io/transitland-lib">transitland-lib software library</a>.</p><p>While developing the Regional GTFS Realtime Feed, our teams wanted to ensure that the process of merging together all the individual input agency feeds into the single set of output endpoints did not introduce any additional data errors. We made us of the same GTFS Realtime validation tooling and test process as we recently described in a <a href="https://www.interline.io/blog/trb-transit-idea-gtfs-realtime-report/">Transportation Research Board report by Interline and our partners at University of South Florida</a>.</p><h2 id="how-to-consume-the-regional-gtfs-realtime-feed">How to consume the Regional GTFS Realtime Feed</h2><p>The Regional GTFS Realtime Feed is now available to all developers through the 511 Open Transit Data API. For instructions on how to access the endpoints and to sign up for an API key, see this <a href="https://groups.google.com/g/511sfbaydeveloperresources/c/VlHlYyj4vRA/m/sLdfHXhdAgAJ">post to the 511 SF Bay Developer Resources mailing list</a>.</p>]]></content:encoded></item><item><title><![CDATA[Our GTFS Realtime validation report published by the Transportation Research Board]]></title><description><![CDATA[In 2019 and 2020, Interline and the Center for Urban Transportation Research at University of South Florida collaborated on a Transit IDEA Program grant from the US Transportation Research Board. Our goal has been to help transit agencies to validate and improve the quality of their GTFS Realtime feeds. This first grant is now complete and TRB has published the full report with our findings and proposed next steps.]]></description><link>https://www.interline.io/blog/trb-transit-idea-gtfs-realtime-report/</link><guid isPermaLink="false">67255750b62841000136ede8</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Tue, 19 Jan 2021 08:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/trb-t-93-report-cover.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/trb-t-93-report-cover.png" alt="Our GTFS Realtime validation report published by the Transportation Research Board"><p>In 2019 and 2020, Interline and the <a href="https://www.cutr.usf.edu/">Center for Urban Transportation Research</a> at University of South Florida collaborated on a <a href="http://apps.trb.org/cmsfeed/TRBNetProjectDisplay.asp?ProjectID=4695">Transit IDEA Program grant</a> from the US Transportation Research Board. Our goal has been to help transit agencies to validate and improve the quality of their GTFS Realtime feeds. This first grant is now complete and TRB has published <a href="http://www.trb.org/Main/Blurbs/181415.aspx">the full report with our findings and proposed next steps</a>.</p><figure><img src="https://www.interline.io/content/images/2024/11/trb-t-93-report-cover-1.png" alt="TRB Transit IDEA T-93 report cover" width="1224" height="1584"><figcaption><p>The full report can be downloaded from the <a href="http://www.trb.org/Main/Blurbs/181415.aspx">TRB website</a>.</p></figcaption></figure><h2 id="executive-summary">Executive Summary</h2><p>Real-time transit information has been shown to have many benefits for transit riders and agencies, including shorter perceived and actual wait times, a more welcoming experience for new riders, and an increased feeling of safety, and increased ridership. Real-time transit data is, in comparison with many other potential operational or capital improvements to bus or rail service, an affordable means of increasing ridership. In the last few years, a real-time complement to the General Transit Feed Specification (GTFS) format, GTFS Realtime, has emerged, which enables transit agencies to share real-time predictions, vehicle positions, and service alert data in a standardized format. Despite its promise, adoption of GTFS Realtime by transit agencies has been hampered by readily available validation tools.</p><figure><img src="https://www.interline.io/content/images/2024/11/gtfs-realtime-architecture-diagram.png" alt="technical architecture diagram showing GTFS Realtime feeds connecting transit agency data systems to consuming applications" width="836" height="842"><figcaption><p>GTFS Realtime feeds (the blueish box on the right) are a critical connection between the complexity of transit agencies' data systems and consuming applications, like smartphone navigation apps.</p></figcaption></figure><p>In this project, Interline Technologies LLC and the Center for Urban Transportation Research (CUTR) at the University of South Florida (USF) have created a prototype platform that makes GTFS Realtime validation tools readily available to, potentially, all transit agencies in North America. Our team is building upon two open-source projects: the <a href="https://github.com/CUTR-at-USF/gtfs-realtime-validator/">GTFS Realtime validator prototype</a> and <a href="https://www.transit.land/">Transitland</a>, an open transit data platform. This project applies the open-source and open-data community models to the challenges of creating and improving GTFS Realtime data.</p><p><strong>Stage 1: Build and Test GTFS Realtime Data Platform</strong>: The research team combined the Transitland open data platform, the GTFS Realtime Validator, and a list of 162 GTFS Realtime feed endpoints (provided by a partner organization). The combined platform collects GTFS Realtime data from each feed, runs the validator process, and produces a report on any detected errors. Each report shows the counts of data entities, the percentage with errors, and a brief text description of any errors. Links take users to additional documentation about each error type. Some errors also provide further contextual information in maps and tables to assist users as they try to determine root causes.</p><figure><img src="https://www.interline.io/content/images/2024/11/transitland-validation-report-screenshots.png" alt="screenshots of the Transitland GTFS Realtime validation reports" width="2000" height="902"><figcaption><p>Screenshots of example GTFS Realtime validation reports. <em>Left</em>: Listing errors (with red severity labels) and a warning (with a yellow severity label) detected in 24 hours of vehicle-position updates. <em>Right</em>: Viewing a specific issue to understand why a vehicle (the little red dot) is far from its assigned route (the blue line).</p></figcaption></figure><p><strong>Stage 2: Testing and Expanding Catalogue</strong>: The project team has tested the platform by preparing validation reports for seven public-transit agencies and reviewing the results in the platform user interface with agency staff members over video calls. In these user-testing sessions, the project team collected information from agency staff about how GTFS and GTFS Realtime data are currently created at each agency, known issues, and any open goals. After being given a tour through the platform and its interface, agency staff reviewed the reports for their own GTFS Realtime feeds. Agency staff were asked to provide input on both the specific quality checks and the overall presentation and approach used by the platform with a standardized question list. The research team has summarized notes from the user-testing sessions and used these results to inform preliminary plans for expanding the platform for a wider range of users in the future.</p><p><strong>Product Pay-Off Potential</strong>: By combining the open-source components of a GTFS Realtime validator with a catalog of GTFS Realtime feeds, hosted on Transitland’s cloud servers, this project will make the process of validating real-time data simple and accessible to agency staff from any computer with a web browser. As a result, GTFS Realtime data will improve in quality and availability. Transit riders will have a better experience (which has been linked to higher ridership), agency staff will provide better service with less effort and cost, and system vendors will provide a higher quality product.</p><p><strong>Product Transfer</strong>: To succeed, this platform will need to be usable by agency staff and to provide them with results that they can act upon, both within the context of their agencies and with their vendors. Therefore, the project team has involved agency stakeholders early and often in the project. The team recruited nine transit agencies to write letters of support as part of the Transit IDEA proposal, and seven agencies have participated in our user-testing process. These agencies represent a diverse range of rider population sizes, staff skill level, and location (urban and rural). The final report includes information on immediate plans to continue and expand Transitland’s catalog GTFS Realtime feeds and longer term plans to develop a GTFS Realtime certification process supported by the platform.</p><h2 id="next-steps">Next Steps</h2><p>We invite everyone to browse Transitland’s <a href="https://www.transit.land/feeds/">list of GTFS Realtime feeds</a>. Please help us to maintain this list by contributing additions and edits to the <a href="https://github.com/transitland/transitland-atlas">open Transitland Atlas feed registry</a>.</p><p>Do you work for a transit agency that publishes GTFS Realtime data? Please check to make sure it’s on Transitland. If not, please add it to the <a href="https://github.com/transitland/transitland-atlas">open Transitland Atlas feed registry</a> or <a href="mailto:info@interline.io">contact us for assistance</a>.</p><p>Do you work for a transit agency that has issues with GTFS Realtime data, or do you represent a group with a stake in the future of GTFS Realtime data? We’re looking to collaborate on opening this GTFS Realtime validation platform for self-serve use by agencies. <a href="mailto:info@interline.io">We welcome your interest</a>.</p><h2 id="acknowledgements">Acknowledgements</h2><p>Research team members include:</p><ul><li><a href="https://www.linkedin.com/in/irees">Ian Rees</a>, Interline, Principal</li><li><a href="https://www.linkedin.com/in/seanbarbeau/">Sean Barbeau</a>, Center for Urban Transportation Research (CUTR), University of South Florida (USF), Principal Mobile Software Architect for R&amp;D</li></ul><p>Thanks to members of the T-93 expert review panel (ERP):</p><ul><li>Patricia Collette</li><li>Stephen M. Stark</li><li>Aaron Antrim</li><li>Carol Schweiger</li><li>Murat Omay</li><li>Santosh Mishra</li></ul><p>Many thanks to Velvet Basemera-Fitzpatrick, Ph.D., Transit IDEA Program Manager.</p><p>Thanks to the following transit agencies and staff members for participating in user testing:</p><ul><li>Metropolitan Transportation Commission (San Francisco): Nisar Ahmed</li><li>Metro Transit (Minneapolis): Laura Matson, Mark DiPasquale, Gary Nyberg, Joey Reid, Juan Villanueva</li><li>Massachusetts Bay Transportation Authority (Boston): Paul Swartz, Jessie, Richards, Logan Nash</li><li>TriMet (Portland, Oregon): Mike Gilligan, Guy Tinat</li><li>Tompkins Consolidated Area Transit (Ithaca, New York): Tom Clavel, Matthew Yarrow</li><li>Washington Metropolitan Area Transit Authority (Washington, D.C.): Stephanie Jones, Richard Carman</li><li>Hillsborough Area Regional Transit (Tampa, Florida): Dexter Corbin, William Mozel, Tim Wictor</li><li>Rogue Valley Transportation District (Medford, Oregon): Melissa Lowry</li></ul>]]></content:encoded></item><item><title><![CDATA[SF Bay Area's Regional GTFS Feed Expanded]]></title><link>https://www.interline.io/blog/mtc-regional-gtfs-feed-additions/</link><guid isPermaLink="false">672547ebb62841000136ed0a</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Mon, 10 Aug 2020 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/headway-july-only.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/headway-july-only.png" alt="SF Bay Area's Regional GTFS Feed Expanded"><blockquote><p><strong>October 4, 2022</strong>: MTC and Interline have released updated fares and transfer discounts for all Bay Area transit agencies using the adopted GTFS-Fares v2 &quot;base implementation.&quot; See <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-fares-updates/">this blog post for more information</a>.</p></blockquote><blockquote><p><strong>September 10, 2021</strong>: MTC and Interline have now released the Regional GTFS <em>Realtime</em> Feed, which provides up-to-the-minute service alerts, trip updates, and vehicle positions using the same ID scheme as the static Regional GTFS Feed. See <a href="https://www.interline.io/blog/mtc-regional-gtfs-realtime-feed/">this blog post for an overview</a>.</p></blockquote><p>Earlier this year, <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-release/">Interline and the Metropolitan Transportation Commission released the Regional GTFS Feed for the San Francisco Bay Area</a>. The Regional GTFS Feed is produced on a daily basis and made available through <a href="https://511.org/open-data">511 SF Bay’s Open Data Portal</a> and its Open Transit Data API.</p><p>We’re pleased to now share a series of additions to the Regional GTFS Feed that we’ve released together over recent months:</p><ul><li><a href="#historical-feeds"><strong>Historical Feeds</strong></a> which provide a retrospective look at an entire month of service</li><li><a href="#station-pathways-and-levels"><strong>Station Pathways and Levels</strong></a> to provide richer wayfinding for all types of riders</li><li><a href="#fares-and-transfer-discounts"><strong>Fares and Transfer Discounts</strong></a> [beta] to calculate the cost of transit journeys on and across the eight largest agencies</li></ul><h2 id="historical-feeds">Historical Feeds</h2><p>Since we released the Regional GTFS Feed in January, much has changed across the Bay Area. Transit agencies and their dedicated staff have rapidly reduced and re-targeted transit service for those riders who perform essential work or depend upon transit for their necessary journeys.</p><p>Using the newly released Historical Feeds component of the Regional GTFS Feed, we can visualize how transit service has changed throughout the Bay Area from January to July of 2020:</p><figure><img src="https://www.interline.io/content/images/2024/11/headway-animation.jpg" alt="" width="2000" height="1582"><figcaption><p>Buses, trains, and ferries throughout the San Francisco Bay Area color-coded by frequency: the reddest route lines are routes that operate at least every 10 minutes; medium hues of red are 10 - 20 minute headways; the lightest hue of red are routes with service less than once every 20 minutes. This animation is created by using the Historical Feeds for the months of January through July of 2020.</p></figcaption></figure><p>Historical Regional Feed products are fully valid GTFS feeds, but they differ somewhat in their contents from the daily Regional Feed products. Read on to understand the process used to produce the Historical Regional Feed products and their key differences, or skip to the end of this section to <a href="#download-historical-feed-products">download Historical Feed products</a>.</p><h3 id="slicing-regional-feeds">Slicing regional feeds</h3><p>Each day, the Regional Feed is produced from the versions of agency feeds on 511.org that provide the best view of service on that day. Each month, the Historical Regional Feed creation process takes these Regional Feeds and combines them together, taking one day of service from each feed, which we are calling a “slice.”</p><p>For example:</p><table><thead><tr><th>Feed filename</th><th>Published</th><th>Contributes service slice for</th></tr></thead><tbody><tr><td>mtc-regional-feed-2020-04-24.zip</td><td>2020-04-24</td><td>2020-04-24</td></tr><tr><td>mtc-regional-feed-2020-04-23.zip</td><td>2020-04-23</td><td>2020-04-23</td></tr><tr><td>mtc-regional-feed-2020-04-22.zip</td><td>2020-04-22</td><td>2020-04-22</td></tr><tr><td>mtc-regional-feed-2020-04-21.zip</td><td>2020-04-21</td><td>2020-04-21</td></tr><tr><td>mtc-regional-feed-2020-04-20.zip</td><td>2020-04-20</td><td>2020-04-20</td></tr></tbody></table><p>If the Regional Feed for a given day is missing, the closest previous day provides service. For instance, if 2020-04-22 was missing, the 2020-04-21 feed slice would cover both 2020-04-21 and 2020-04-22.</p><h3 id="global-entity-copying">Global entity copying</h3><p>Agencies, stops, and routes are considered “global”, and are handled using a simple ID-based merge with the most recent version winning. For example, if BART has a route with ID “OR-S” that is called “Richmond - Warm Springs”, but then later renames it to “Richmond to Warm Springs”, then the latter version will be used.</p><h3 id="trip-hashing-comparison-and-copying">Trip hashing, comparison, and copying</h3><p>Trips are more complicated and handled separately. A simple combining of all the trips and stop_times in all of the input files can easily create a GTFS feed that is too large for practical use, especially given that programs like <a href="https://www.interline.io/routing-platform/">OpenTripPlanner</a> need to hold the entire schedule in memory. Therefore, duplicate copies of trips are detected using a hash based approach and only copied to the output once. This reduces the output size by approximately 90%.</p><p>For example, here are three hypothetical versions of Trip ID “BA:2210503” from three consecutive days of input regional feeds.</p><table><thead><tr><th>Feed filename</th><th>Trip ID</th><th>Route ID</th><th>Service ID</th><th>Headsign</th><th>1st stop</th><th>2nd stop</th><th>n stops</th><th>Hash</th></tr></thead><tbody><tr><td><code>2020-04-24.zip</code></td><td><code>BA:2210503</code></td><td><code>BA:OR-S</code></td><td><code>BA:Wkd_BASE-Weekday-07</code></td><td>Warm Springs/South Fremont</td><td><code>RICH 5:03</code></td><td><code>DELN 5:07</code></td><td><code>PLZA 5:10...</code></td><td><code>8c4ecb</code></td></tr><tr><td><code>2020-04-23.zip</code></td><td><code>BA:2210503</code></td><td><code>BA:OR-S</code></td><td><code>BA:Wkd_BASE-Weekday-07</code></td><td>Warm Springs/South Fremont</td><td><code>RICH 5:03</code></td><td><code>DELN 5:07</code></td><td><code>PLZA 5:10...</code></td><td><code>8c4ecb</code></td></tr><tr><td><code>2020-04-22.zip</code></td><td><code>BA:2210503</code></td><td><code>BA:OR-S</code></td><td><code>BA:Wkd_BASE-Weekday-07</code></td><td>Warm Springs to South Fremont</td><td><code>RICH 5:04</code></td><td><code>DELN 5:06</code></td><td><code>PLZA 5:12...</code></td><td><code>a4bf1a</code></td></tr></tbody></table><p>For each of these, the hashing function takes into account all trip attributes, all the calendar attributes for that trip, and the full details of each entry in <code>stop_times.txt</code>. Any change in any field will result in a different hash. This allows us to directly compare trips between versions of the input feed. Above, all details and schedule for 2020-04-23 and 2020-04-24 match exactly, so these trips two will be considered identical. The trip for 2020-04-22 has some minor differences in name and schedule, so will generate a different hash, and be considered a different trip.</p><p>As the historical feed merging program processes each input feed, it calculates the hash of each trip in the feed. If it has not seen a trip before, it copies it to the output and notes the hash for future use. If it has been seen before, it is not copied again. To prevent clashes, the original Trip IDs are appended with the trip hash (e.g. <code>BA:2210503</code> -&gt; <code>BA:2210503:8c4ecb</code>). The merging program then takes all trips in the input feed (both seen and unseen) and examines the calendars to see which are active for each day in this slice, and then creates <code>calendar_dates.txt</code> entries for each trip on each day where that trip is scheduled to run. The original service IDs are changed to be the same as the hash appended Trip ID, and the calendar is unrolled into a day-by-day format, but it works reliably. This hashing approach is resource efficient and allows us to create historical feeds of arbitrary duration while minimizing the output size.</p><p>Example output <code>calendar_dates.txt</code>:</p><table><thead><tr><th>Service ID</th><th>Date</th><th>Exception Type</th></tr></thead><tbody><tr><td><code>BA:2210503:8c4ecb</code></td><td>2020-04-24</td><td>1 (Added)</td></tr><tr><td><code>BA:2210503:8c4ecb</code></td><td>2020-04-23</td><td>1 (Added)</td></tr><tr><td><code>BA:2210503:a4bf1a</code></td><td>2020-04-22</td><td>1 (Added)</td></tr></tbody></table><p>In this way, the <code>8c4ecb</code> version of the trip is scheduled to run on the two days of input data where it was seen, and the <code>a4bf1a</code> version is scheduled to run on the other day.</p><h3 id="differences-between-regional-and-historic-feeds">Differences between Regional and Historic feeds</h3><p>Historic Regional Feeds are equivalent to the original daily Regional Feeds in the stops, routes, and scheduled services they contain. Using a Historic will produce the same output in a routing engine or another type of analysis.</p><p>Historic Feeds are different from Regional Feeds in their specific GTFS structure:</p><ul><li><code>calendars.txt</code> records are removed and rewritten in <code>calendar_dates.txt</code></li><li><code>trips.txt</code> records are hashed and compared (as described above)</li><li>IDs for global records are namespaced (as described above)</li></ul><p>These differences should not affect routing engine or similar types of analysis. However, keep these differences in mind if you are trying to use historical feeds to understand changes in GTFS data and its practices over time at Bay Area agencies.</p><h3 id="download-historical-feed-products">Download Historical Feed products</h3><p>To use the Historical Feed products:</p><ol><li><a href="https://511.org/open-data/token">Sign up for a 511 Open Data API token</a></li><li>Download from <code>http://api.511.org/transit/datafeeds?api_key=[your_key]&amp;operator_id=RG&amp;historic=YYYY-MM</code> (for example, to request May 2020: <code>historic=2020-05</code>)</li><li>At the start of each month, the Historic Feed for the last month is created and posted. You may download as many months as you wish and combine them to analyze as many months/quarters/years as you wish at once.</li></ol><p>As of this blog post, the following months are available for download (that is, you can you any of these values for the <code>historic</code> query parameter):</p><ul><li><code>2020-01</code></li><li><code>2020-02</code></li><li><code>2020-03</code></li><li><code>2020-04</code></li><li><code>2020-05</code></li><li><code>2020-06</code></li><li><code>2020-07</code></li></ul><h2 id="station-pathways-and-levels">Station Pathways and Levels</h2><p>The Regional Feed exists to both merge together individual agency GTFS feeds and to serve as a home for new GTFS data that describes cross-agency conditions. The daily Regional Feed now comes with layouts for 35 key transit stations across the Bay Area, each of which serves multiple agencies. Using the newly added <code>pathways.txt</code> and <code>levels.txt</code> files in GTFS, Interline and MTC are now able to provide more detailed information about how to transfer from one agency to another. This includes information on elevators, escalators, and routes that may not be accessible to those in wheelchairs. Pathways and levels information will equip trip planning apps to provide more helpful wayfinding information to their users, particular those who are new to the Bay Area or who have limited vision.</p><figure><img src="https://www.interline.io/content/images/2024/11/university-ave-pathways.jpg" alt="" width="2000" height="948"><figcaption><p>Pathways and levels throughout the University Ave/Downtown Palo Alto station served by Caltrain regional rail on two platforms and VTA and SamTrans buses at bus bays.</p></figcaption></figure><h3 id="_35-regional-transit-hubs">35 Regional Transit Hubs</h3><p>Here is a list of the 35 stations currently in the Regional Feed:</p><table><thead><tr><th>Transit station/hub</th><th>County</th></tr></thead><tbody><tr><td>Embarcadero BART</td><td>San Francisco</td></tr><tr><td>Montgomery BART</td><td>San Francisco</td></tr><tr><td>Caltrain Station 4th &amp; King</td><td>San Francisco</td></tr><tr><td>Salesforce Transit Center</td><td>San Francisco</td></tr><tr><td>12th St Oakland City Center BART</td><td>Alameda</td></tr><tr><td>El Cerrito Del Norte BART</td><td>Contra Costa</td></tr><tr><td>19TH St Oakland BART</td><td>Alameda</td></tr><tr><td>Powell ST BART</td><td>San Francisco</td></tr><tr><td>Civic Center BART</td><td>San Francisco</td></tr><tr><td>Walnut Creek BART</td><td>Contra Costa</td></tr><tr><td>Richmond BART/Amtrak</td><td>Contra Costa</td></tr><tr><td>San Jose Diridon Station</td><td>Santa Clara</td></tr><tr><td>Palo Alto Station</td><td>Santa Clara</td></tr><tr><td>San Rafael Transit Center</td><td>Marin</td></tr><tr><td>Millbrae BART</td><td>San Mateo</td></tr><tr><td>Pleasant Hill BART</td><td>Contra Costa</td></tr><tr><td>San Francisco Ferry Terminal</td><td>San Francisco</td></tr><tr><td>Daly City BART</td><td>San Francisco</td></tr><tr><td>Santa Rosa Transit Mall</td><td>Sonoma</td></tr><tr><td>Union City BART</td><td>Alameda</td></tr><tr><td>MacArthur BART</td><td>Alameda</td></tr><tr><td>Dublin/ Pleasanton BART</td><td>Alameda</td></tr><tr><td>Warm Springs/ South Fremont BART</td><td>Alameda</td></tr><tr><td>Santa Clara Caltrain</td><td>Santa Clara</td></tr><tr><td>Oakland Coliseum BART</td><td>Alameda</td></tr><tr><td>SFO</td><td>San Francisco</td></tr><tr><td>Fairfield Transportation Center</td><td>Solano</td></tr><tr><td>OAK</td><td>Alameda</td></tr><tr><td>Petaluma Transit Mall</td><td>Marin</td></tr><tr><td>Vallejo Ferry Terminal</td><td>Solano</td></tr><tr><td>Mountain View Station</td><td>Santa Clara</td></tr><tr><td>Great America</td><td>Santa Clara</td></tr><tr><td>Napa Intermodal</td><td>Napa</td></tr><tr><td>SJC</td><td>Santa Clara</td></tr><tr><td>Great Mall/Milpitas BART</td><td>Santa Clara</td></tr></tbody></table><p>Note that just as the ongoing pandemic has made our work on the Historic Feed all the more relevant, our work on Station Pathways and Levels has had to adapt to current circumstances. Our staff have been working from home using aerial imagery, architectural drawings, station maps, and other materials. Interline’s Station Editor tool also works on tablets, and we look forward to again going out into the field to all of these stations to correct and improve details.</p><p>We welcome questions and corrections sent to the <a href="https://groups.google.com/forum/#!forum/511sfbaydeveloperresources">511SFBayDeveloperResources mailing list</a>.</p><h3 id="using-station-pathways-and-levels">Using Station Pathways and Levels</h3><p>Download the daily Regional Feed like so:</p><ol><li><a href="https://511.org/open-data/token">Sign up for a 511 Open Data API token</a></li><li>Download from <code>http://api.511.org/transit/datafeeds?api_key=[your_key]&amp;operator_id=RG</code></li><li>Look for the <code>pathways.txt</code> and <code>levels.txt</code> files inside the zip archive.</li><li>For more information, see the <a href="https://gtfs.org/reference/static#pathwaystxt">static GTFS documentation</a> and the <a href="https://bit.ly/gtfs-pathways">GTFS-Pathways extension proposal document</a>.</li></ol><h2 id="fares-and-transfer-discounts">Fares and Transfer Discounts</h2><p>To date, the Regional GTFS Feed has focused on how to plan a journey by transit. Now we’re curating and adding additional data to help riders understand how to pay for their journeys. The Bay Area’s Regional GTFS Feed is now the first in the world to include data using the GTFS Fares-v2 specification. This <a href="https://bit.ly/gtfs-fares">newly proposed specification</a> allows us to model the many different fare products and discounts that are available to the Bay Area’s transit riders. For example, we can now account for how some agencies provide riders a discount when they pay by Clipper Card rather than by cash. (<a href="https://www.clippercard.com/">Clipper Card</a> is the contact-less payment card and system operated by MTC and available on buses, trains, and ferries throughout the Bay Area.) We can also capture how riders can receive a discount when transferring from certain agencies to other agencies. Read on for a detailed example of how Fares-v2 work in practice, or skip ahead to <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-additions/#using-fares-and-transfer-discounts">use the Fares-v2 beta data</a>.</p><h3 id="an-example-of-fares-v2-in-practice">An Example of Fares-v2 in Practice</h3><figure><img src="https://www.interline.io/content/images/2024/11/bart-to-ac-transit-route.png" alt="map of a journey by BART and AC Transit originating in San Francisco, transferring in Oakland, and ending in Alameda" width="1606" height="1042"><figcaption><p>A journey by BART and AC Transit, from San Francisco to Oakland to Alameda, which we will use in the following example of how to calculate the total cost. (Credit: BART Trip Planner)</p></figcaption></figure><p>Fares-v2 expands the traditional route- and zone-based GTFS fares model with several additional files, each focused on modeling a different part of a complex fare scheme. Imagine a rider making a transfer from BART to AC Transit; both BART and AC Transit have different prices for adult fares, eligible discount fares, and different prices when paying with cash or when using a Clipper Card; additionally, the discount applied when transferring from BART to AC Transit has different values and rules when using Clipper. Calculating the individual fares for each leg of this trip requires a description of any zones that apply to routes and stops (<code>fare_networks.txt</code> and <code>fare_areas.txt</code>), the costs and requirements of the base fare for each leg (<code>fare_leg_rules.txt</code>), any discount categories that might include the rider (<code>fare_profiles.txt</code>), and the type of payment used by the rider (<code>fare_containers.txt</code>). All of these factors are considered and then used to search for any transfer discounts that may apply for the trip (<code>fare_transfer_rules.txt</code>).</p><p>Let’s work through the hypothetical trip above, first taking BART from Embarcadero Station to 12th St. in Oakland, and then riding an AC Local bus, 51A, to Alameda.</p><p>The following is an except from <code>fare_leg_rules.txt</code> with some columns removed for brevity. This file contains sets of rules that are matched against a single leg of a trip, without regard to transfers. The columns <code>from_area_id</code>, <code>to_area_id</code>, <code>fare_container_id</code> and <code>fare_category_id</code> references values defined in the other files mentioned above.</p><table><thead><tr><th>order</th><th>leg_group_id</th><th>from_area_id</th><th>to_area_id</th><th>amount</th><th>fare_container_id</th><th>fare_category_id</th><th>notes</th></tr></thead><tbody><tr><td>90</td><td>BA:</td><td>EMBR</td><td>12TH</td><td>1.35</td><td>clipper</td><td>BA:Senior/Disabled Clipper</td><td>Embarcadero to 12th with Clipper and discount</td></tr><tr><td>100</td><td>BA:</td><td>EMBR</td><td>12TH</td><td>3.70</td><td>clipper</td><td></td><td>Embarcadero to 12th with Clipper</td></tr><tr><td>110</td><td>BA:</td><td>EMBR</td><td>12TH</td><td>4.20</td><td></td><td></td><td>Embarcadero to 12th with cash surcharge</td></tr></tbody></table><p>The BART leg would match the first the rules above based on <code>from_area_id</code> and <code>to_area_id</code>, with fares of $1.35, $3.70, and $4.20. However, the first two rules are only available when using a Clipper Card (<code>fare_container_id</code>), and the first of these is only available to riders eligible for a discounted fare (<code>fare_category_id</code>). The <code>order</code> field is used to select the correct fare when multiple rules match the trip leg: any applicable discounted Clipper fare would match first, followed by adult Clipper fare, then finally the fare including additional charge for cash riders. In the event two (or more) rules match with the same order value, both are considered valid options to present to the rider.</p><table><thead><tr><th>order</th><th>leg_group_id</th><th>from_area_id</th><th>to_area_id</th><th>amount</th><th>fare_container_id</th><th>fare_product_id</th><th>fare_category_id</th><th>notes</th></tr></thead><tbody><tr><td>100</td><td>AC:local</td><td>AC:local</td><td>AC:local</td><td>2.25</td><td>clipper</td><td></td><td></td><td>Adult local fare with clipper $2.25</td></tr><tr><td>110</td><td>AC:local</td><td>AC:local</td><td>AC:local</td><td>2.5</td><td></td><td></td><td></td><td>Adult local fare $2.5</td></tr><tr><td>100</td><td>AC:local</td><td>AC:local</td><td>AC:local</td><td>1.12</td><td></td><td></td><td>AC:senior</td><td>Discounted local fare $1.12 with Clipper</td></tr><tr><td>100</td><td>AC:local</td><td>AC:local</td><td>AC:local</td><td>1.25</td><td></td><td></td><td>AC:senior</td><td>Discounted local fare $1.25</td></tr><tr><td>100</td><td>AC:local</td><td>AC:local</td><td>AC:local</td><td>0</td><td></td><td>AC:local:monthly</td><td></td><td>Free local fare with monthly pass</td></tr></tbody></table><p>The AC Transit fare rules are similar, with the addition of a few extra columns. The 51A is an East Bay local only route (<code>to_area_id</code> is assumed), and matches each rule, and the rules for discounts and cash are applied as above. The <code>fare_product_id</code> field describes additional rules that are available to users of certain prepaid products, such as day passes and monthly passes. In this case, a rider with an Adult Local 31 Day Pass ($84, described in <code>fare_products.txt</code>) would enjoy a free ride. More complex AC Transit fares use additional columns not pictured here (<code>fare_network_id</code>) and handle cases such as riding a Transbay bus and the different prices applied when riding within the East Bay vs. taking the trip all the way to the Salesforce Transit Center in downtown San Francisco.</p><p>Any applicable transfers are calculated by reading the rules in <code>fare_transfer_rules.txt</code> and matching each leg of the trip against each subsequent leg of the trip. This can create quite complicated models, but let’s start by looking at our BART and AC Transit legs. The clever bit is that many <code>fare_leg_rules.txt</code> rules can share the same <code>leg_group_id</code>, which simplifies the number and types of transfer rules that must be defined. For example, all of the BART rules above use <code>BA:</code> and all of the AC Transit rules use <code>AC:local</code>. The table below is an except of <code>fare_transfer_rules.txt</code>.</p><table><thead><tr><th>order</th><th>from_leg_group_id</th><th>to_leg_group_id</th><th>fare_container_id</th><th>amount</th><th>duration_limit</th><th>duration_limit_type</th><th>fare_transfer_type</th><th>spanning_limit</th><th>notes</th></tr></thead><tbody><tr><td>100</td><td>BA:</td><td>AC:local</td><td>clipper</td><td>-0.5</td><td>90</td><td>2</td><td>1</td><td>2</td><td>1 credit of $-0.50 to AC w/in 90 minutes</td></tr><tr><td>110</td><td>BA:</td><td>AC:local</td><td></td><td>-0.25</td><td></td><td></td><td>1</td><td>3</td><td>Cash: 2 fare credits of $0.25 each</td></tr></tbody></table><p>Two different discounts apply when taking an AC Transit leg after a BART leg. When using Clipper, a $0.50 (<code>amount=-0.5</code>, <code>fare_transfer_type=1</code>) discount is applied to the first AC Transit leg (<code>spanning_limit=2</code>, which means it can only match a subjourney of two legs). Alternatively, a paper transfer slip given to cash users contains two tabs, each of which applies a $0.25 discount to the AC Transit cash fare (<code>spanning_limit=3</code>, or a subjourney of up to three legs). Additionally, the Clipper discount expires within 90 minutes after tagging off BART (<code>duration_limit=90</code>, <code>duration_limit_type=2</code>). Because the <code>order</code> value is different, these two transfer options are mutually exclusive to Clipper and cash riders respectively.</p><p>The combination of matching fare rules and fare transfer rules allows describing even complicated transfers, such as discounts to SamTrans local bus riders who hold a Caltrain 2 zone or higher monthly pass, the several categories of upgrade charges when transfering from an AC Transit Local to an AC Transit Transbay bus, and the many varied products and transfers that apply full or partial fare to trips on the Dumbarton Express.</p><h3 id="agencies-in-the-beta-release">Agencies in the Beta Release</h3><p>Note that this is a “beta” release. The GTFS Fares-v2 specification is still being finalized. It does not fully capture some functionality, like how AC Transit and VTA use “fare capping” so that riders never have to pay more in one day than the cost of a day pass. We also expect that as trip planners and other apps begin to consume the Regional Feed’s fare data that we may need to revise some of the data or our schema.</p><p>To our knowledge, Interline is the only organization to have a “rules engine” that can calculate journey costs using GTFS Fares-v2 data. We expect that as other organizations adopt the Interline fares rules engine or build their own equivalents, we may need to revisit some of the assumptions built into the engine.</p><p>The beta release of data includes fares and transfer discounts within and between eight agencies:</p><ul><li>BART</li><li>SFMTA</li><li>AC Transit</li><li>SamTrans</li><li>VTA</li><li>Caltrain</li><li>Golden Gate Transit (bus only; not ferries)</li><li>Dumbarton Express</li></ul><p>This list includes the seven largest agencies and a key connector across San Francisco Bay (Dumbarton Express). It should provide a broad enough sample of fares and transfer discounts to power a wide range of potential applications. This sample is also deep enough to inform analyses of fares, the cost of riding transit, and ideally even ways to reform fares and transfer discounts to produce more equitable outcomes for Bay Area transit riders.</p><h3 id="using-fares-and-transfer-discounts">Using Fares and Transfer Discounts</h3><p>Download the daily Regional Feed like so:</p><ol><li><a href="https://511.org/open-data/token">Sign up for a 511 Open Data API token</a></li><li>Download from <code>http://api.511.org/transit/datafeeds?api_key=[your_key]&amp;operator_id=RG</code></li><li>For more information about the fares and transfer discount files in the GTFS feed, see the <a href="https://bit.ly/gtfs-fares">GTFS-Fares v2 proposal document</a>.</li></ol><p>Interline and MTC expect to improve both the Fares-v2 beta data and schemas as we learn more. We welcome questions and corrections sent to the <a href="https://groups.google.com/forum/#!forum/511sfbaydeveloperresources">511SFBayDeveloperResources mailing list</a>.</p><h2 id="public-transit-in-2020">Public Transit in 2020</h2><p>A final note that behind every piece of transit data and every multi-modal trip plan are real people who drive vehicles, serve riders, and maintain equipment and facilities. Thank you to all the front-line workers keeping the Bay Area’s public transit systems and riders moving safely!</p><h2 id="acknowledgements">Acknowledgements</h2><p>Credit and many thanks to project team members including Ian Rees and Ruth Miller (Interline), Nisar Kapeel and Kapeel Daryani (MTC), and our partners at Bay Area transit agencies.</p>]]></content:encoded></item><item><title><![CDATA[A Regional GTFS Feed for the San Francisco Bay Area]]></title><description><![CDATA[Last year, Interline began a "startup in residence" at the Metropolitan Transportation Commission. Now we're pleased to publicly release our first creation: the San Francisco Bay Area's Regional GTFS Feed.]]></description><link>https://www.interline.io/blog/mtc-regional-gtfs-feed-release/</link><guid isPermaLink="false">6725531fb62841000136ed64</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Wed, 08 Jan 2020 08:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/mtc-regional-feed-all-routes.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/mtc-regional-feed-all-routes.png" alt="A Regional GTFS Feed for the San Francisco Bay Area"><blockquote><p><strong>September 10, 2021</strong>: MTC and Interline have now released the Regional GTFS <em>Realtime</em> Feed, which provides up-to-the-minute service alerts, trip updates, and vehicle positions using the same ID scheme as the static Regional GTFS Feed. See <a href="https://www.interline.io/blog/mtc-regional-gtfs-realtime-feed/">this blog post for an overview</a>.</p></blockquote><blockquote><p><strong>August 10, 2020</strong>: MTC and Interline have released additional functionality in the Regional GTFS Feed. See <a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-additions/">this more recent blog post</a>.</p></blockquote><p>Last year, <a href="https://www.interline.io/blog/metropolitan-transportation-commission-selects-interline/">Interline began a “startup in residence” at the Metropolitan Transportation Commission</a>. Now we’re pleased to publicly release our first creation together: the San Francisco Bay Area’s Regional GTFS Feed.</p><p>Every day, Interline’s workflow processes 31 agency feeds and creates a single unified static GTFS feed.</p><figure><img src="https://www.interline.io/content/images/2024/11/mtc-regional-feed-animation.jpg" alt="" width="1530" height="1990"><figcaption><p>Transit routes around the Bay Area, first by agency feed, then from the aggregated Regional GTFS Feed.</p></figcaption></figure><h2 id="why-a-regional-feed">Why a Regional Feed?</h2><p>People don’t stop their commutes or their travels at transit agency boundaries. Similarly, many applications require combining multiple agency feeds: trip planners, visualizations, data analyses, travel demand models, etc. <strong><em>The Regional GTFS Feed will provide data consumers an easier option.</em></strong> Less effort to ingest multiple feeds, more time toward your actual goals.</p><p><strong><em>The Regional GTFS Feed enables the addition of cross-agency data</em></strong>: fares, transfer discounts, other transfer information, and transit hub layouts and accessibility information. These types of data reference IDs and entities from multiple agencies and are a natural fit within a consolidated feed.</p><p>The workflow that creates the Regional GTFS Feed also identifies conflicts between agency feeds, data entities, and IDs. <strong><em>By validating agency feeds together, the workflow identifies data quality issues that are only apparent once individual feeds are combined.</em></strong> We’re sharing this information with MTC and agency staff to help further improve all feeds.</p><p>Data consumers are still able to access individual feeds from agency websites and <a href="https://511.org/developers/list/apis/">511.org developer APIs</a>. The Regional GTFS Feed is produced using a “conservative” approach: data transformation rules are targeted and constrained. Our motto is <em>Merge. Don’t lose.</em></p><h2 id="future-plans">Future Plans</h2><p>In 2020, we’re continuing to work with MTC to further enrich the Regional GTFS Feed:</p><ul><li>We’re <strong><em>mapping key transit hubs</em></strong> around the Bay Area, creating GTFS-Pathways data using Interline’s new Gotransit editing tools.<a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-release/#footnote-1">*</a> This will help trip planning websites and apps to provide better directions to those who use wheelchairs or have other mobility limitations.</li><li>We’re <strong><em>cataloging fares, tickets, and transfer discounts around the Bay Area</em></strong>, creating GTFS-Fares v2 data, also using Interline’s Gotransit editing tools.<a href="https://www.interline.io/blog/mtc-regional-gtfs-feed-release/#footnote-2">**</a> Transit fares and discounts across agencies is a problem of combinatorial complexity. The Regional GTFS Feed will not be able to precisely capture every possible fare, pass, and transfer discount. The goal is to capture sufficient detail to create realistic fare estimates for a variety of rider types riding and transferring between the agencies of highest ridership.</li><li>To start, the Regional GTFS Feed is updated once per day. In the future, we’ll also <strong>create historical versions of the Regional GTFS Feed</strong> that combine “slices in time” from different agency feed versions. While trip planning is best powered by the latest GTFS feed, many types of analyses and travel models instead require archival feed versions from the past.</li></ul><p>Please watch 511.org’s <a href="https://511.org/open-data/transit">Open Transit Data</a> section for updates as these additions are released.</p><h2 id="powered-by-transitland">Powered by Transitland</h2><p>The Regional GTFS Feed has much in common with <a href="https://www.transit.land/">Transitland</a>, our open data platform that aggregates GTFS data for over 2,500 agencies around the world. We’ve used many of the lessons we’ve learned from Transitland to create the Regional GTFS Feed workflow. In fact, the workflow is build using the core, open-source ingredients of <a href="https://www.transit.land/news/2019/10/17/tlv2.html">Transitland version 2</a> (also known as “Tlv2”) The 30+ agency feeds are represented as a DMFR (a Distributed Mobility Feed Registry) and processed using the Gotransit library. The Bay Area’s Regional GTFS Feed workflow is, in effect, a Transitland of its own, which is exposed using a static GTFS export.</p><h2 id="to-use-the-feed-and-ask-questions">To Use the Feed and Ask Questions</h2><p>To use the new Regional GTFS Feed:</p><ol><li><a href="https://511.org/open-data/token">Sign up for a 511 Open Data API token</a></li><li>Download from <code>http://api.511.org/transit/datafeeds?api_key=[your_key]&amp;operator_id=RG</code></li><li>The feed is updated daily; download it as frequently as your application needs updates.</li></ol><p>To provide feedback or ask questions, please visit the <a href="https://groups.google.com/forum/#!forum/511sfbaydeveloperresources">511SFBayDeveloperResources</a> Google group.</p><h2 id="acknowledgements">Acknowledgements</h2><figure><img src="https://www.interline.io/content/images/2024/11/mtc-logo.png" alt="Metropolitan Transportation Commission logo" width="516" height="128"></figure><p>Thanks to our partners in MTC’s Technology Services division. They’ve been serving transit data through 511.org since before GTFS and GTFS Realtime specifications. It’s a pleasure learning from their experience and working together to take advantage of the latest advances in transit data.</p><figure><img src="https://www.interline.io/content/images/2024/11/stir-banner-1.png" alt="Startup in Residence logo" width="231" height="128"></figure><p>Thanks to also to <a href="https://www.cityinnovate.com/">City Innovate</a> for managing the Startup in Residence (STIR) program. We encourage procurement officers and civic-oriented startups to learn more about the program.</p><h2 id="notes">Notes</h2><p><a id="footnote-1"></a></p><p><sup>*</sup> For more on the GTFS-Pathways additions, see <a href="http://bit.ly/gtfs-pathways">this Google doc</a>.</p><p><a id="footnote-2"></a></p><p><sup>**</sup> For more on the GTFS-Fares-v2 proposal, see <a href="http://bit.ly/gtfs-fares">this Google doc</a>.</p>]]></content:encoded></item><item><title><![CDATA[Even more geospatial tools supporting the "GeoJSONL" format]]></title><description><![CDATA["GeoJSONL" has the easy readability of a GeoJSON file without the the performance limitations of the format. Last year we started producing GeoJSONL files from our OSM Extracts service. Now a growing list of long-existing and brand-new geospatial tools.]]></description><link>https://www.interline.io/blog/here-cli-support-geojsonl/</link><guid isPermaLink="false">6725410eb62841000136ec69</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 08 Mar 2019 08:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/download-geojsonl-extract.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/download-geojsonl-extract.png" alt="Even more geospatial tools supporting the &quot;GeoJSONL&quot; format"><p>Last year we expanded our <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> service to produce what we call GeoJSONL. This is a data format that goes by many names: newline-delimited GeoJSON, line-oriented GeoJSON, GeoJSONSeq, or GeoJSON Text Sequences. Whatever its name, the format enables effficient reading and efficient writing of <a href="https://en.wikipedia.org/wiki/GIS_file_formats#Vector">vector geo-data</a> features.</p><p>For more background, see our <a href="https://www.interline.io/blog/geojsonl-extracts/">previous blog post on GeoJSONL</a> and see this new <a href="https://stevage.github.io/ndgeojson/">overview of the format’s many names and its defining properties</a>.</p><p>Over the last year, there’s been even more progress on tools for writing and reading GeoJSONL (or whatever your preferred name is for this format).</p><h2 id="geojsonl-support-in-here-xyz">GeoJSONL support in HERE XYZ</h2><p>HERE XYZ now supports GeoJSONL ingest.</p><p>Don’t know what HERE XYZ is? Here’s how we summarized the platform in a <a href="https://www.interline.io/blog/here-xyz-open-geo-data/">blog post</a> last year:</p><blockquote><p>HERE XYZ provides the full set of components that &quot;modern&quot; geo developers have come to expect: GeoJSON input and output, &quot;slippy map&quot; libraries, a RESTful API, a JavaScript SDK, a cross-platform CLI, base maps and styles. As we've been experimenting with the HERE XYZ beta, we've found it provides some unique combinations that make it especially well suited to combining open geodata APIs — especially in large quantities.</p></blockquote><p>Using the latest version of the <a href="https://www.npmjs.com/package/@here/cli">HERE CLI</a>, you can parse and upload GeoJSONL files to the HERE XYZ platform.</p><p>Here’s an update of our <a href="https://interline-io.github.io/here-xyz-tutorials/tutorial-1/">“Use Crowdsourced Data from Interline OSM Extracts in an XYZ Web Map” tutorial</a> that uses the GeoJSONL format.</p><p>Download a GeoJSONL extract from OSM Extracts:</p><figure><img src="https://www.interline.io/content/images/2024/11/download-geojsonl-extract-1.png" alt="terminal showing usage of HERE CLI to upload OSM Extract in GeoJSONL format" width="2000" height="1085"></figure><p>Upload to XYZ and set tags:</p><figure><img src="https://www.interline.io/content/images/2024/11/here-cli-upload-geojsonl.png" alt="screenshot of a terminal window" width="1688" height="502"></figure><p>Previously you might hit memory limits on your computer if you tried to upload an extract for a big metro region. Now, thanks to GeoJSONL’s efficient parsing, you should be able to upload any extract of any size!</p><h2 id="the-growing-toolchain">The growing toolchain</h2><p>There’s a long history of tools that work with this format, and it’s been growing over the last year. Here are some more highlights:</p><p><strong>Command line</strong> On the command line, read <code>.geojsonl</code> and <code>.geojsons</code> files using GDAL’s <a href="https://gdal.org/drivers/vector/geojsonseq.html"><code>drv_geojsonseq</code></a>.</p><p><strong>Desktop GIS</strong> GDAL also powers <a href="https://www.qgis.org/">QGIS</a>, so now you can load <code>.geojsonl</code> and <code>.geojsons</code> files into our favorite desktop GIS sofware.</p><p><strong>Parse in Python</strong>: <a href="https://github.com/Toblerity/Fiona">Fiona</a> is a Python library that wraps GDAL and provides a “neat and nimble” interface. The next release will include <a href="https://github.com/Toblerity/Fiona/issues/720">support</a> for <code>drv_geojsonseq</code>.<a href="https://www.interline.io/blog/here-cli-support-geojsonl/#footnote-1">*</a></p><p><strong>Statistics and data science in R</strong> If you’re loading vector geo-data into R, try the <a href="https://cran.r-project.org/package=geojson">geojson</a> package on CRAN. See these <a href="https://github.com/ropensci/geojson#newline-delimited-geojson">instructions</a> for how to load a <code>.geojsonl</code> file directly from Interline OSM Extracts.</p><p><strong>Convert from a shapefile</strong> Install the <a href="https://www.npmjs.com/package/shapefile">shapefile NPM package</a> and run <code>shp2json --newline-delimited</code> to convert a shapefile to GeoJSONL. See these <a href="https://github.com/mbostock/shapefile#shp2json_newline_delimited">instructions</a>.</p><p><strong>Convert to vector tiles</strong> <a href="https://github.com/mapbox/tippecanoe/">Tippecanoe</a> can take GeoJSONL as input and turn it in to <a href="https://github.com/mapbox/awesome-vector-tiles">MVT vector tiles</a>. It also provides its own <code>tippecanoe-json-tool</code> command to convert a standard GeoJSON file to GeoJSONL format (see <a href="https://github.com/mapbox/tippecanoe/#tippecanoe-json-tool">these instructions</a>).</p><p><strong>Stream in to a web browser</strong> Watch a GeoJSONL file stream into a web in this <a href="https://observablehq.com/@bcamper/streaming-geojsonl">interactive JavaScript notebook</a>.<a href="https://www.interline.io/blog/here-cli-support-geojsonl/#footnote-2">†</a> If you have an Interline OSM Extracts API key and an Observable account, try <a href="https://observablehq.com/@drewda/streaming-geojsonl-from-your-choice-of-openstreetmap-extr">this notebook</a> to select from any of the 200+ cities and watch the stream arrive:</p><figure><img src="https://www.interline.io/content/images/2024/11/streaming-geojsonl.jpg" alt="" width="600" height="547"></figure><h2 id="when-to-try-geojsonl">When to try GeoJSONL</h2><p>If you like GeoJSON but find your data files are too big to hold in memory, give one of these tools a try.</p><p>If you need to run a bunch of vector geo-data through a <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> operation that, give this format a try. (It’s easy to divide a GeoJSONL file in to chunks before parsing its exact contents.)</p><p>If you want to download <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> from Interline and explore even the largest of extracts using HERE XYZ, give this a try.</p><hr><h3 id="notes">Notes</h3><p><a id="footnote-1"></a></p><p><sup>*</sup> Shoutout to <a href="https://github.com/sgillies">Sean Gillies</a>, the lead creator of Fiona and instigator of <a href="https://tools.ietf.org/html/rfc8142">RFC 8142 - GeoJSON Text Sequences</a>. We may have differing views on the <a href="https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text">record separator character</a>. Regardless, his packages like Fiona and <a href="https://rasterio.readthedocs.io/">Rasterio</a> are fabulous and we’re happy users of them.</p><p><a id="footnote-2"></a></p><p><sup>†</sup> Shoutout to <a href="http://vector.io/">Brett Camper</a>, the author of this Observable notebook and a fellow alum of <a href="https://mapzen.com/">Mapzen</a>.</p>]]></content:encoded></item><item><title><![CDATA[Transportation Research Board funds an open platform for transit agencies to improve the quality of their real-time data]]></title><description><![CDATA[We’re excited to share that the Transportation Research Board has awarded a Transit IDEA Program grant to Interline. We’re using this unique opportunity to expand Transitland to support GTFS Realtime feeds.]]></description><link>https://www.interline.io/blog/transportation-research-board-funds-gtfs-realtime/</link><guid isPermaLink="false">67255705b62841000136eddc</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Wed, 23 Jan 2019 08:00:00 GMT</pubDate><content:encoded><![CDATA[<blockquote><p><strong>January 19, 2021</strong>: Interline and CUTR@USF have now published a Transportation Research Board report on this project. See <a href="https://www.interline.io/blog/trb-transit-idea-gtfs-realtime-report/">this more recent blog post</a>.</p></blockquote><p>We’re excited to share that the <a href="http://www.trb.org/">Transportation Research Board</a> has awarded a <a href="http://apps.trb.org/cmsfeed/TRBNetProjectDisplay.asp?ProjectID=4695">Transit IDEA Program grant</a> to Interline. We’re using this unique opportunity to expand <a href="https://www.transit.land/">Transitland</a> to support <a href="https://developers.google.com/transit/gtfs-realtime/">GTFS Realtime</a> feeds. Our goal is for Transitland to provide a suite of validation tools for public-transit agency staff to validate the contents and quality of their real-time data feeds.</p><h2 id="high-quality-real-time-data-is-hard-but-worthwhile">High-quality real-time data is hard but worthwhile</h2><figure><img src="https://www.interline.io/content/images/2024/11/trb-idea-the-problem.png" alt="the problem statement" width="960" height="720"></figure><p>Real-time transit information has many benefits to transit riders and agencies, including shorter perceived and actual wait times, a more welcoming experience for new riders, and an increased feeling of safety, and increased ridership. Real-time transit data is, in comparison with many other potential operational or capital improvements to bus or rail service, an affordable means of increasing ridership. In the last few years, a real-time complement to the General Transit Feed Specification (GTFS) format, GTFS Realtime, has emerged.</p><p>Despite its promise, adoption of GTFS Realtime by transit agencies has been hampered by a lack of clear documentation and readily available validation tools. Hardware and software vendors for automatic vehicle location (AVL) systems also vary in the depth and quality of their support for the GTFS Realtime specification. As a result, transit agencies must invest significant time and effort to create and maintain high quality GTFS Realtime feeds. Furthermore, bad data has been shown to have a negative effect on ridership, the rider’s opinion of the agency, and the rider’s satisfaction with multimodal trip-planning apps. Thus, transit agencies must put even more effort toward ensuring that the GTFS Realtime data produced by their systems is of sufficient quality.</p><h2 id="our-solution-gtfs-validation-in-transitland">Our Solution: GTFS Validation in Transitland</h2><p>In this project, Interline and the <a href="https://www.cutr.usf.edu/">Center for Urban Transportation Research</a> are creating a prototype platform that makes GTFS Realtime validation tools readily available to, potentially, all transit agencies in North America. We will build upon two open-source projects: the <a href="https://github.com/CUTR-at-USF/gtfs-realtime-validator">GTFS Realtime validator prototype</a> and Transitland, an open transit data platform. This project will apply the open-source and open-data community models to the challenges of creating and improving GTFS Realtime data.</p><p>By combining the open-source components of a GTFS Realtime validator with a catalog of GTFS Realtime feeds, hosted on Transitland’s cloud servers, this project will make the process of validating real-time data simple and accessible to agency staff from any computer with a web browser. As a result, GTFS Realtime data will improve in quality and availability. Transit riders will have a better experience (which has been linked to higher ridership), agency staff will provide better service with less effort and cost, and system vendors will provide higher quality product.</p><h2 id="our-partner-center-for-urban-transportation-research">Our Partner: Center for Urban Transportation Research</h2><p>We’re glad to have on our project team an expert on GTFS Realtime: Sean Barbeau. Sean and his colleagues at the University of South Florida’s Center for Urban Transportation Research have been involved in GTFS, GTFS Realtime, and related transit data specifications and systems for many years. Most recently, they have created a GTFS Realtime validator prototype. We’ll be working together to test and deploy this validator tooling within the Transitland platform.</p><p>For more about their preliminary findings about the quality of GTFS Realtime feeds, see <a href="https://medium.com/@sjbarbeau/introducing-the-gtfs-realtime-validator-e1aae3185439">this recent summary of their TRB paper</a>. Here are the most common errors and warnings their prototype validator found in a sample of GTFS Realtime feeds:</p><figure><img src="https://www.interline.io/content/images/2024/11/validator-test-errors.png" alt="most common errors found in a sample of GTFS Realtime feeds" width="624" height="431"><figcaption><p>Image Credit: Sean Barbeau, Transportation Research Board 2018 paper <a href="https://trid.trb.org/view/1496848">18–05585 &quot;Quality Control — Lessons Learned from the Deployment and Evaluation of GTFS-realtime Feeds&quot;</a>.</p></figcaption></figure><h2 id="about-the-transit-idea-program">About the Transit IDEA program</h2><p>Here’s an overview of the <a href="http://www.trb.org/IDEAProgram/IDEATransit.aspx">Transit IDEA Program</a>:</p><blockquote><p>The Transit IDEA Program is part of the Transit Cooperative Research Program, a cooperative effort of the Federal Transit Administration (FTA), the Transportation Research Board (TRB), and the Transit Development Corporation (a nonprofit educational and research organization of the American Public Transportation Association). The program is funded by the FTA and is managed by TRB.</p><p>The Transit IDEA Panel has established four high-priority focus areas to encourage proposals in the following areas:Increasing transit ridershipImproving transit safety, security, and emergency preparednessImproving transit capital and operating efficienciesProtecting the environment and promoting energy independence.</p></blockquote><p>IDEA stands for “Innovations Deserving Exploratory Analysis” — which we think is an apt description for the Transitland platform providing GTFS Realtime feed validation to agency staff!</p><h2 id="an-open-call-to-transit-agency-staff">An open call to transit agency staff</h2><p>If your agency provides a GTFS Realtime feed, we welcome you to participate in the user-testing phase of this project. Please write us at <a href="mailto:info@interline.io">info@interline.io</a> for more information.</p>]]></content:encoded></item><item><title><![CDATA[Transitland Version 2]]></title><description><![CDATA[What first started as two simple, separate, little experiments has grown into the seeds of a major revision of the entire Transitland platform: Transitland Version 2.]]></description><link>https://www.interline.io/blog/tlv2/</link><guid isPermaLink="false">67255876b62841000136ee13</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Tue, 08 Jan 2019 08:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/tlv2-architecture-diagram.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/tlv2-architecture-diagram.png" alt="Transitland Version 2"><blockquote><p><strong>March 4, 2021</strong>: Here's a more recent blog post <a href="https://www.interline.io/blog/transitland-v2-search/">demonstrating Transitland v2's new ability to search across transit operators, feeds, and routes</a>.</p></blockquote><blockquote><p><strong>February 25, 2021</strong>: Here's a more recent blog post <a href="https://www.interline.io/blog/transitland-v2-vector-map-tiles/">demonstrating Transitland v2's new global transit map and vector tiles</a>.</p></blockquote><p>It's been a quiet year on this blog, and we're overdue to share all the work that's been going on behind the scenes since the <a href="https://www.interline.io/blog/transitland-continues/">last post</a>. What first started as two simple, separate, little experiments has grown into the seeds of a major revision of the entire Transitland platform: Transitland Version 2.</p><h2 id="tlv2-functionality-and-goals">Tlv2 Functionality and Goals</h2><p><em>Tlv2</em> (as we've come to call this new version) provides continuity on the most used parts of the Transitland platform. Tlv2 will still provide a canonical service at transit.land that includes a Feed Registry of open GTFS data feeds from around the world, a queryable API, and a range of user interfaces. Tlv2, like Tlv1, will be powered by open-source components. Tlv2's API will continue to be accessible without charge, and most of the Tlv1 API will continue to be available during the transition. (As before, we will enforce rate limits to ensure no one user hogs too many server resources; more on how rate limits will change below.)</p><p>What we are changing in Tlv2 is in support of three goals:</p><ul><li>Higher Performance</li><li>More Modular Architecture</li><li>Distributed Deployments</li></ul><h2 id="goal-1-higher-performance">Goal #1: Higher Performance</h2><p>Tlv1 successfully parses and imports GTFS feeds from thousands of transit agencies. However its &quot;FeedEater&quot; component slows down or halts on a handful of the very larger GTFS feeds — typically single feeds that cover entire countries. This has always been a constraint on Tlv1.</p><p>Interline's new <code>transitland-lib</code> (tlib) library solves this problem. Transitland-lib is a new transit data library written in the <a href="https://golang.org/">Go programming language</a>, which provides finer-grained control of memory usage, as well as much easier patterns for concurrency and parallelization of complex tasks. Transitland-lib replaces replaces FeedEater, and will be used for feed validation, pre-processing, and importing data into the Tlv2 database. Transitland-lib is efficient enough to handle the largest feeds in a small memory footprint, and is fast enough to encourage interactive use; imports that took hours with FeedEater are completed in minutes.</p><p>Transitland-lib began as an experiment by Ian Rees over the last winter holidays, and it now complete enough to power multiple Interline projects, from the San Francisco Bay Area Regional GTFS Feed (which merges ~30 feeds into one) to a transit-to-routing-engine pipeline that Interline built for another client (which covers transit service for an entire country!). We look forward to releasing transitland-lib under an open-source license shortly.</p><p>Transitland's most demanding API users have also hit performance limitations over the years. As part of Tlv2, we'll be providing bulk data dumps that give users another option. Use the Tlv2 API to explore data or power a small web app; use Tlv2 data extracts to power a large analysis, a giant web app, or a country-wide routing engine.</p><h2 id="goal-2-more-modular-architecture">Goal #2: More Modular Architecture</h2><p>Tlv1 has succeeded in part due to its &quot;monolith&quot; architecture. FeedEater, the API, and administrative operations all live within the same <a href="https://www.transit.land/documentation/datastore/">Transitland Datastore</a> codebase. This has allowed for efficient reuse of data models, helpers, and test code. The single Datastore codebase is also straightforward to deploy and configure for the canonical transit.land site. The Datastore codebase can be reused — some have in fact redeployed it for their own regional or internal GTFS management needs—but customizing and redeploying the Datastore is not straightforward.</p><p>For Tlv2, we're breaking the Transitland Datastore into modular components. Some components will be in Go (data processing), some will remain in Ruby (general back-end) and JavaScript (user interfaces). The canonical transit.land deployment will continue to use all. Other deployments by Interline, our partners, or open-source users may only use a subset. This more modular architecture will also allow deployments that connect Tlv2 components to internal data pipelines, routing engines, and other tools that may not be relevant to the canonical transit.land deployment.</p><p>While the software components of Tlv2 will be more modular, they will still be glued together using the data schemes developed for Transitland, including <a href="https://www.transit.land/documentation/concepts/onestop-id-scheme">Onestop IDs</a> and a common feed format spec (described below). Additionally, we will be sharing our database schema in versions appropriate for PostgreSQL and SQLite to help simplify additional application development, including outside of the canonical transit.land.</p><h2 id="goal-3-distributed-deployments">Goal #3: Distributed Deployments</h2><p>The Feed Registry at transit.land will continue to be our canonical list of GTFS feeds. However, some deployments of Tlv2 components may want to use just a subset of the Feed Registry or a private list of feeds. We also want to open up the canonical Feed Registry for more collaborative tending, beyond just staff from Interline and Trillium Transit.</p><p>The second of the two experiments that has led to Tlv2 is called the <a href="https://github.com/transitland/distributed-mobility-feed-registry">Distributed Mobility Feed Registry</a>. DMFR (for short) is a standard way of representing a list of GTFS feeds as one or more JSON files. It also currently supports GTFS-RT, and we expect it to extend to GBFS (bike rental) and MDS (e-scooter rental) in the future.</p><p>(Some history: This experiment actually started way back in the early tests of Transitland, before the fully built out Tlv1 monolith. It's still <a href="https://github.com/transitland/transitland-datastore">archived</a> on GitHub. Now we've come full circle to revisit and build out this approach.)</p><p>Soon the <code>transitland/distributed-mobility-feed-registry</code> repository on GitHub will become the source of truth for the Transitland Feed Registry of public open data feeds. We'll accept pull requests against the JSON files to add or edit feeds. This should substantially lower the burden for outside contributors to contribute new feeds and help manage existing records. However, this will be just be one repository of DMFR files. In effect we're blowing up the single Transitland Feed Registry and inviting the creation of many.</p><p>Transitland-lib can manage a datastore using one or more DMFR files, meaning that the canonical transit.land, distributed instances of Tlv2, and temporary interactive sessions with transitland-lib on the command line can all share the same representation for mobility data feeds. At Interline, we've gone one step further and also use DMFR files to configure some of our routing engines and any other workflow that takes two or more mobility data feeds as input. We welcome other consumers or transformers of GTFS, GTFS-RT, and related data to try a DMFR file for their own needs.</p><h2 id="tlv2-architecture-diagram">Tlv2 Architecture Diagram</h2><p>Now that we've introduced and discussed the new components of Tlv2, here's a diagram of how they connect:</p><figure><img src="https://www.interline.io/content/transit-land/tlv2/tlv2-architecture-diagram.png" alt="Tlv2 architecture diagram" width="768" height="942"></figure><h2 id="gradual-deprecation-of-tlv1">Gradual Deprecation of Tlv1</h2><p>Tlv1 requires a fair amount of attention and resources to maintain. While developing Tlv2, we've already had to split our attention. As we shift our focus to deploying and supporting Tlv2, we'll be gradually deprecating Tlv1. Don't worry — it won't be too quickly.</p><p><strong>Step 1: Tlv2 API Preview</strong> The initial release of Tlv2 will include a preview of a new <a href="https://graphql.org/">GraphQL</a> API that provides flexible, structured access to transit data. All new data will be available through this API from the start. API access will require a key. Sign-up will be free. We will institute tight rate limits at first and increase them over time, as we better understand the type and complexity of queries that users run.</p><p><strong>Step 2: Deprecate Tlv1 Schedule API</strong> Immediately upon Tlv2 release, we will stop importing new <a href="https://www.transit.land/documentation/datastore/schedules.html">schedule data</a> into Tlv1. The <code>ScheduleStopPair</code> data model is the most expensive part of Tlv1 to run and maintain. We'll first stop imports, keeping the API available against the last imported version. We'll share tutorials on how to switch to the Tlv2 GraphQL API to access schedules. Then, after sufficient notice, we'll empty old Tlv1 schedule data and remove that API endpoint.</p><p><strong>Step 3: Other Tlv1 Entity API Endpoints</strong> While the Tlv2 GraphQL API will make a number of complex queries much simpler than with the current Tlv1 API, it may take a while for us to make all Tlv1 query options available in the Tlv2 API. During this time, we'll try to continue to power the Tlv1 APIs (with the exception of the schedules endpoint). We'll recommend that you build any new applications on the Tlv2 API, in case we do have to announce any specific incompatibilities during the Tlv1-to-Tlv2 API transition.</p><p>We'll post updates to this site. Or watch <a href="https://twitter.com/transitland">@transitland</a> and <a href="https://twitter.com/interline_io">@interline_io</a> for updates on Twitter.</p><h2 id="how-you-can-help">How You Can Help</h2><p>Here are ways you can help with the rollout of Tlv2:</p><ul><li>Provide your input on Tlv2 by taking our <a href="https://docs.google.com/forms/d/e/1FAIpQLSdmhW28zcP1kBMBkmelAt7OQGqKjRcoILjsbZvGXLsGfmFNGw/viewform">survey and signing up for the beta tester list</a>.</li><li>Comment on the <a href="https://funding.communitybridge.org/projects/transitland">DMFR GitHub repository</a>. Contribute edits once we migrate over the Feed Registry to DMFR files.</li><li>If you work for a GTFS producing transit operator or mobility provider, <a href="mailto:info@interline.io">contact Interline</a> to learn more about &quot;Tlv2 on premise&quot; and regional GTFS feeds. It's consulting projects like this that help us serve both mobility providers and further the development of transitland-lib.</li><li>Contribute financially to the upkeep and growth of Tlv2 through the Linux Foundation's <a href="https://funding.communitybridge.org/projects/transitland">CommunityBridge donation page for Transitland</a>. These contributions go directly toward upkeep of the canonical transit.land.</li></ul>]]></content:encoded></item><item><title><![CDATA[Transitland Continues in 2019]]></title><description><![CDATA[Just like a garden, open-source and open-data projects require continual tending. With 2019 beginning, we're glad to share more widely these plans for Transitland's ongoing tending and growth as an open platform.]]></description><link>https://www.interline.io/blog/transitland-continues/</link><guid isPermaLink="false">672559d3b62841000136ee31</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Tue, 08 Jan 2019 08:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/transitland_logo.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/transitland_logo.png" alt="Transitland Continues in 2019"><p>Just like a garden, open-source and open-data projects require continual tending. During 2018, we quietly maintained the core of Transitland's code and data. We also worked to put in place new partnerships and structures to support Transitland after the <a href="https://mapzen.com/blog/shutdown/">shuttering of Mapzen</a>. With 2019 beginning, we're glad to share more widely these plans for Transitland's ongoing tending and growth as an open platform.</p><h2 id="core-support-and-maintenance">Core Support and Maintenance</h2><p>We have a combination of organizations that are together supporting Transitland:</p><ul><li>Transitland's open-source code is moving to the <a href="https://www.linuxfoundation.org/">Linux Foundation</a>, allowing code use and contributions by any and all.</li><li>Amazon Web Services continues to provide servers and access for Transitland's API users, thanks to awards from the <a href="https://aws.amazon.com/earth/research-credits/">Earth on AWS Cloud Credits for Research Program</a>.</li><li><a href="https://www.interline.io/">Interline Technologies</a> (the firm that I am now a part of) has been donating resources to maintain code and is now also offering professional services and support to the companies that depend on Transitland APIs.</li><li><a href="https://trilliumtransit.com/">Trillium Solutions</a>, a team of experts at GTFS production, is helping to curate submissions to the Transitland Feed Registry and maintain existing records.</li></ul><p>Previously Mapzen was a single &quot;keystone&quot; supporter for Transitland. Now we have a growing list of &quot;tentpole&quot; supporters for Transitland's code, servers, and data. Together in 2018, we quietly shifted Transitland on to a new generation of infrastructure, fixed pressing bugs, and gradually increased feed coverage. Transitland now aggregates data for 2,466 transit operators across 53 countries! Each month, the API handles 5 million queries!</p><figure><img src="https://www.interline.io/content/transit-land/transitland-continues/transitland-operators-by-country-2019.png" alt="bar chart of Transitland operators by country" width="257" height="1158"></figure><p><em>Operators records in Transitland, by</em> <a href="https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2"><em>two-letter country code</em></a><em>.</em></p><p>Now that Transitland again has a stable foundation of support, post Mapzen, we can return to feature development, major data additions, and community outreach.</p><h2 id="growth-and-development-in-2019">Growth and Development in 2019</h2><p>We're working together with a series of partners and sponsors on new Transitland platform features and data additions in 2019. The list of topics tentatively includes:</p><ul><li><a href="https://www.interline.io/blog/transportation-research-board-funds-gtfs-realtime/">GTFS-realtime support</a></li><li>feed validation and best-practices report cards</li><li><a href="https://www.interline.io/blog/metropolitan-transportation-commission-selects-interline/">feed merging</a></li><li>&quot;Transitland Tiles&quot; for bulk download and analysis</li><li>website and documentation updates</li><li>an experimental, distributed replacement to the Transitland Feed Registry</li></ul><p><strong>Technical folks</strong>: As you've likely guessed, many of these new experiments require a level of performance that Transitland's current mix of scripting languages cannot always provide. My colleague <a href="https://www.interline.io/about/">Ian Rees</a>, who is Interline's principal engineer, is leading an effort to re-architect key Transitland components in the high-performance Go programming language. Transitland's technical architecture has always had the goal of accessibility — now we're pursuing both accessibility <em>and</em> performance.</p><p><strong>Professional users of Transitland APIs</strong>: We've heard from many of you in 2018 that new functionality is nice, but that dependable service is even more important. Interline is now preparing professional services and support around the Transitland APIs. <a href="http://eepurl.com/dmWHln">Sign up for the Interline newsletter</a> for related announcements or contact <a href="mailto:info@interline.io">info@interline.io</a> with questions. Note that these commercial services supplement and do not change existing free access to the Transitland APIs.</p><p><strong>All Transitland Community Members</strong>: Thanks for sticking with us over the past year. Despite all the work happening behind the scenes, we weren't able to share full news of Transitland's progress publicly. We're excited to return to our previous cadance of public announcements and conversations. Watch this website for updates. We also welcome questions and participation from potential partners in the above topics and others — please write to <a href="mailto:hello@transit.land">hello@transit.land</a>.</p><p>Here's to a new year of open transit data!</p>]]></content:encoded></item><item><title><![CDATA[Bringing Back the Spirit of Map Mashups: Combine Open Geodata APIs using HERE XYZ]]></title><description><![CDATA[Interline and HERE are collaborating to make Interline's open data services like OSM Extracts and the Transitland API easy to combine, mix, filter, analyze, and visualize using the new HERE XYZ platform. ]]></description><link>https://www.interline.io/blog/here-xyz-open-geo-data/</link><guid isPermaLink="false">6725453eb62841000136ecbc</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Mon, 01 Oct 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/xyz-tutorial-1-animation.jpg" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/xyz-tutorial-1-animation.jpg" alt="Bringing Back the Spirit of Map Mashups: Combine Open Geodata APIs using HERE XYZ"><p>At Interline, we spend much of our time with open geodata APIs. We create our own open geodata APIs (like <a href="https://www.transit.land/">Transitland</a>) and we use others (for example, whenever we need to throw a bunch of GeoJSON on a base map). At their best, open geodata APIs are fun — you get to combine pieces together to create something new. Then again, public APIs can have many limitations — combine together a few and you can end up with the worst of all worlds: duplicate data, misregistered overlays, a slow user experience, incompatible data formats… With all these mixed experiences in our minds, we’ve been playing with the initial beta of <a href="http://explore.xyz.here.com/">HERE XYZ</a>. It’s a brand new developer service from one of the oldest names in geodata.</p><p>HERE XYZ provides the full set of components that “modern” geo developers have come to expect: GeoJSON input and output, “slippy map” libraries, a RESTful API, a JavaScript SDK, a cross-platform CLI, base maps and styles. As we’ve been experimenting with the HERE XYZ beta, we’ve found it provides some unique combinations that make it especially well suited to combining open geodata APIs — especially in large quantities.</p><p>Combining geodata is nothing new. <a href="http://ladprofile.weebly.com/ian-mcharg-1920-2001.html">Ian McHarg</a>, a prominent environmental designer, printed maps on transparent plastic sheets and overlaid them atop each other, inspiring the idea of layers in GIS (<a href="https://en.wikipedia.org/wiki/Geographic_information_system">geographic information systems</a>):</p><figure><img src="https://www.interline.io/content/images/2024/11/ian-mcharg.jpg" alt="photo of Ian McHarg" width="1100" height="742"></figure><p><em>Image credit: University of Pennsylvania</em></p><p>More recently, Google Maps brought us <em>mashups</em>. At first, it involved hacking the Google Maps JavaScript code in order to overlay <a href="https://www.nytimes.com/2005/12/11/magazine/doityourself-cartography.html">crimes in Chicago</a> and <a href="http://www.housingmaps.com/">Craigslist house listings</a>:</p><figure><img src="https://www.interline.io/content/images/2024/11/housingmaps2.png" alt="screenshot of old housingmaps.com" width="1088" height="684"></figure><p><em>Image credit: housingmaps.com</em></p><p>Neither hacking nor reverse-engineering are required any more to combine multiple sources of geodata in a web browser. Companies like Google, Esri, Mapbox, Carto — also a now defunct start-up where we previously worked, called Mapzen — all offer accessible APIs for creating web maps. We’ve also been freed of many of the constraints on dataset size in 2005 mashups. Original mashups involved drawing points and other geographic features on top of base-map images, with a low limit on the number of these vector features that could be drawn at once. More recent techniques send both the base map and overlays as vector tiles. Instead of an image, a vector tile is effectively a square of data, which can be pre-generated by a server. Many vector tiles are then loaded and drawn by your web browser simultaneously.</p><p>We’ve left behind the constraints of mashups — and it also feels like we’ve forgotten the fun of mashups. Vector tiles help greatly with performance, but often add complexity to the overall set of steps required to combine together existing and new data sources on a web map or another type of visualization.</p><p>Our experiments with the HERE CLI and the XYZ API are bringing back that creative spirit of 2005 mashups. The XYZ platform generates vector tiles near instantly for any new imported data sources. The HERE CLI and XYZ API provide a range of ways to import GeoJSON files, connect to APIs, or stream in from other CLI tools. It’s the best of 2005 and 2018. If you use the SDKs to style your web maps to look like the transparency overlays in Ian McHarg’s book <em>Design With Nature</em>, it’s the best of 1969 proto-GIS, too.</p><p>We’re collaborating with the HERE XYZ team to turn our experiments into tutorials that demonstrate how Interline’s open geodata services can be combined, mixed, filtered, analyzed, and visualized. In the first tutorial, you’ll fetch and filter data from <a href="https://app.interline.io/osm_extracts/interactive_view">Interline OSM Extracts</a>:</p><figure><img src="https://www.interline.io/content/images/2024/11/xyz-tutorial-1-animation-2.jpg" alt="" width="918" height="756"></figure><p>In the second tutorial, you’ll query the <a href="https://www.transit.land/">Transitland</a> API repeatedly to build up an analysis and a web map of just not subway locations but also the frequency of how often each subway line operates. (Especially when working with geodata for transportation applications, both space and time matter.) We’ll show you how to do this for the entire planet:</p><figure><img src="https://www.interline.io/content/images/2024/11/xyz-tutorial-2-animation-3.jpg" alt="" width="800" height="526"></figure><p>The sample projects and tutorials take intermediate and advanced users on a tour through some of the unique features of XYZ and the HERE CLI:</p><ul><li>filtering Interline OSM Extracts using OpenStreetMap tags</li><li>writing scripts that stream geodata from external APIs into an XYZ space using the HERE CLI</li><li>using multiple API queries to aggregate together properties about the same features (identified by unique IDs, such as <a href="https://www.transit.land/documentation/concepts/onestop-id-scheme/">Transitland Onestop IDs</a>)</li><li>quickly building global tile sets</li></ul><p>We’re experimenting with additional features, for creating a new generation of mashups, and we look forward to sharing more soon.</p><p>Now, get started with our <a href="https://www.interline.io/docs/here-xyz/">first two tutorials on combining HERE XYZ, Interline OSM Extracts, and Transitland</a>.</p>]]></content:encoded></item><item><title><![CDATA[Practices that help complex and collaborative open-source projects survive and thrive]]></title><description><![CDATA[We've encountered many practices that can help complex and collaborative open-source projects survive and thrive. This blog post provides a tour through many open-source practices, particularly ones that are helpful for aging but popular software packages like OpenTripPlanner.]]></description><link>https://www.interline.io/blog/open-source-best-practices-for-otp-summit/</link><guid isPermaLink="false">6725557cb62841000136edb3</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 06 Jul 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/table-of-contents.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/table-of-contents.png" alt="Practices that help complex and collaborative open-source projects survive and thrive"><p>At Interline, we specialize in deploying, creating, and managing the growth of open-source software. It’s less an individual matter of principle than a set of shared advantages: our clients build on each other’s advances, and we collaborate productively with a wide range of partners. However, the health of an open-source project is not a given — it requires ongoing attention and thought. In this blog post, we’ll share an overview of practices used by a wide variety of open-source projects:</p><p><strong>Useful practices</strong></p><ul><li>👍 <a href="#clean-licenses">Clean licenses</a></li><li>👍 <a href="#technical-steering-committees">Technical steering committees</a></li><li>👍 <a href="#rfc-request-for-comments-processes">RFC (request for comments) processes</a></li><li>👍 <a href="#feature-flags">Feature flags</a></li><li>👍 <a href="#contrib-vs-core">“Contrib” vs. core</a></li><li>👍 <a href="#automated-acceptance-test-and-performance-suites">Automated acceptance test and performance suites</a></li><li>👍 <a href="#automated-dependency-checks">Automated dependency checks</a></li><li>👍 <a href="#release-management">Release management</a></li><li>👍 <a href="#codes-of-conduct">Codes of conduct</a></li></ul><p><strong>Unclear practices</strong></p><ul><li>🤷 <a href="#the-roadmap">“The roadmap”</a></li></ul><p><strong>Bad outcomes</strong></p><ul><li>👎 <a href="#the-eternal-rewrite">“The eternal rewrite”</a></li><li>👎 <a href="#the-big-switchover">“The big switchover”</a></li></ul><h2 id="opentripplanner-an-open-source-project-going-on-10-years">OpenTripPlanner: An open-source project going on 10 years</h2><figure><img src="https://www.interline.io/content/images/2024/11/otp-logo.png" alt="OpenTripPlanner logo" width="339" height="373"></figure><p>This blog post comes out of a presentation Interline gave at last month’s OpenTripPlanner Summit in Boston. None of the practices are particular to that project. However, some background will help explain how relevant many of these open-source practices are to OpenTripPlanner and similar open-source projects that have become critical — but also somewhat neglected — pieces of infrastructure for many organizations.</p><p>The OpenTripPlanner routing engine (also known as OTP) <a href="http://docs.opentripplanner.org/en/latest/History/">was created in 2009</a> by TriMet (the public-transit agency in Portland, Oregon), OpenPlans (a civic tech non-profit), and individual experts and enthusiasts. Since then, OTP has grown to be used by public-transit agencies, planning firms, and transportation researchers <a href="http://docs.opentripplanner.org/en/latest/Deployments/">around the world</a>. (Interline itself provides managed hosting of OTP for a variety of public-transit agencies around North America.)</p><p>And yet the OTP project rests on an unstable foundation. Many individuals and organizations contribute to developing new features and expanding OTP’s reach, but few are involved directly in maintenance, coordination, and the other “housekeeping chores” of an open-source project.</p><p>One outcome of last month’s OpenTripPlanner Summit is that more organizations will be collaborating on this “housekeeping.” Interline is joining the <a href="http://docs.opentripplanner.org/en/latest/Governance/">OTP Project Leadership Committee</a>, alongside Cambridge Systematics, Conveyal, TriMet, Ruter (the public-transit agency of Oslo), the University of South Florida, the PlannerStack Foundation, and the Helsinki Regional Transport Authority.</p><p>Now, with this context in mind, let’s zoom out and consider a wide range of practices that complex and collaborative open-source projects use to grow and thrive.</p><h2 id="clean-licenses">Clean licenses</h2><h3 id="choosing-a-single-license">Choosing a (single) license</h3><p>At the core of an open-source project is its license. This is what legally enables use and contribution to the project by many individuals and organizations. The Open Source Initiative <a href="https://opensource.org/licenses">catalogs hundreds of licenses and highlights a short list of the most popular options</a>:</p><figure><img src="https://www.interline.io/content/images/2024/11/os-licenses.png" alt="screenshot of list of open-source licenses" width="963" height="779"></figure><p>Different licenses can encourage different outcomes. OpenTripPlanner uses the LGPLv3 license, which requires that any organization directly customizing OTP code publicly share their changes. However, the LGPLv3 license does allow organizations to use OTP within the context of larger proprietary systems.</p><h3 id="using-dual-licenses">Using dual licenses</h3><p>Some projects are released under dual licenses. For example, the Sidekiq project is available for free under LGPLv3. Organizations that require more flexibility (for example, to customize code and keep their changes private), may pay the primary creator of Sidekiq for an alternative license:</p><figure><img src="https://www.interline.io/content/images/2024/11/sidekiq-dual-licenses.png" alt="the Sidekiq project is available under two licenses, shown in this screenshot" width="1220" height="1482"></figure><p>The dual license model can have trade-offs. It may provide resources to build a business around an open-source project, and it may provide a predictable means for buying support for large users. However, it can be off-putting or at least confusing to smaller users who assume “open-source = completely free.” Clear communication is important. Consider the case of the ExtJS project, which <a href="https://en.wikipedia.org/wiki/Ext_JS#License_history">switched to a dual license in 2008</a>, alienating many of its users.</p><h3 id="contributor-license-agreements">Contributor license agreements</h3><p>The presence of a license file in an open-source code repository is a meaningful signal, but it isn’t always sufficent reassurance to businesses that the code is free of legal risk. Some open-source project sponsors ask all contributors to sign a <a href="http://contributoragreements.org/">contributor agreement</a> before their code is merged into the open-source project. Typically, these agreements ask the “signer” to say that they personally created whatever code it is that they are contributing and that they are not knowingly contributing materials for which they don’t hold rights. Some contributor agreements go further, asking contributors to sign over intellectual property rights to the primary company that maintains the project. At a minimum, such an agreement provides all companies involved with some reassurance that contributors aren’t intentionally adding others’ code or private materials.</p><p>The Linux Foundation no longer recommends broad contributor agreements; instead they offer the simple <a href="https://developercertificate.org/">Developer Certificate of Origin</a>.</p><p>Open-source projects that do need to enforce one of these agreements can add a “bot” to GitHub or a similar source-code repository. Here’s <a href="https://github.com/apps/dco">a bot to enforce the Developer Certificate of Origin</a> and <a href="https://cla-assistant.io/">a bot to enforce a custom contributor license agreement</a>:</p><figure><img src="https://www.interline.io/content/images/2024/11/dco-app.png" alt="screenshot: the Developer Certificate of Origin is enforced using this GitHub app" width="674" height="1182"></figure><p>Code licenses and contributor license agreements are necessary but not sufficient for an open-source project to succeed. They set the incentives and the bounds for collaboration and growth.</p><h2 id="technical-steering-committees">Technical steering committees</h2><p>Licenses allow contribution, but who will accept the contributions? Many projects appoint a technical steering committee (or TSC) to make such decisions.</p><p>For example, here are the <a href="https://github.com/nodejs/TSC#list-of-tsc-responsibilities">responsibilities of the Node.js TSC</a>:</p><ul><li><strong>Managing code and documentation creation and changes</strong> for the listed projects and resources</li><li><strong>Setting and maintaining standards</strong> covering contributions of code, documentation and other materials</li><li><strong>Managing code and binary releases</strong>: types, schedules, frequency, delivery mechanisms</li><li>Making decisions regarding dependencies of the Node.js Core project, - including what those dependencies are and how they are bundled with source code and releases</li><li><strong>Creating new repositories and projects</strong> under the nodejs GitHub organization as required</li><li><strong>Setting overall technical direction</strong> for the Node.js Core project, - including high-level goals and low-level specifics regarding features and functionality</li><li><strong>Setting and maintaining appropriate standards for community discourse</strong> via the various mediums under TSC control</li><li><strong>Setting and maintaining governance rules</strong> for the conduct and make-up of the TSC, Working Groups and other bodies within the TSC's domain</li></ul><p>The <a href="https://en.wikipedia.org/wiki/Node.js#History">history of the Node.js TSC</a> is useful to consider. Development of Node.js had nominally been overseen by a single company, named Joyent. When outside contributors to the project found their contributions and their specific technical needs not addressed by Joyent staff, <a href="https://www.javaworld.com/article/2855639/open-source-tools/qanda-why-io-js-decided-to-fork-node-js.html">they forked the Node.js code</a> (that is, they copied the code) and started a new effort. This new project, called io.js, was formed around a TSC and an “open” governance process. After much negotiation, Joyent and the io.js TSC combined their efforts under the newly formed Node.js Foundation. The code improvements made by this “splinter” effort found their way back into the Node.js code base. More importantly, the new governance structure continued.</p><p>Consider another example: the OneBusAway project. It’s governed by <a href="https://onebusaway.org/the-onebusaway-project/project-members/">a steering committee</a> that is structured to include a mix of representatives from the private sector, the public sector, universities, and independent individuals.</p><p>As these two examples demonstrate, the primary challenge of a TSC is ensuring that it has an appropriate representation across the open-source project’s contributors and users. If a TSC has too few members or members that are only concerned with low-level software development concerns, the project will miss opportunities to learn from less technically sophisticated users. Alternatively, with a TSC that is too broad and includes individuals or organizations that are not deeply invested in the project, the risk is that the TSC will have many ideas but little action.</p><h2 id="rfc-request-for-comments-processes">RFC (request for comments) processes</h2><p>Technical steering committees (TSCs) can put thoughts into actions by various means. Let’s look at the RFC (that is, request for comments) process now. Later on, we’ll also consider the ”<a href="https://www.interline.io/blog/open-source-best-practices-for-otp-summit/#the-roadmap">roadmap</a>“.</p><p>The Internet and its protocols <a href="https://en.wikipedia.org/wiki/Request_for_Comments#History">were created and continue to mature using RFCs</a>. RFCs are documents that put into words all of the ingredients of a proposed technical change. They are both speculative and precise. They are always pedantic (many RFCs begin by defining exactly what the authors mean by “must” and “must not”) and sometimes humorous (well, humorous in the eyes of fellow engineers). An RFC is a structured format for proposing new software development and honing plans based on feedback.</p><p>The Rust programming language has <a href="http://rust-lang.github.io/rfcs/">a well defined RFC process</a>; it’s been used as a model by many other open-soure projects. For the Rust project, contributors must write an RFC for:</p><ul><li>Making “Substantial changes” that are not bug fixes</li><li>Removing existing features</li><li>Changing interfaces between components</li></ul><p>RFCs are not required for:</p><ul><li>Refactoring, reorganizing</li><li>“Additions that strictly improve objective, numerical quality criteria (warning removal, speedup, better platform coverage, more parallelism, trap more errors, etc.)”</li><li>“Additions only likely to be noticed by other developers-of-rust, invisible to users-of-rust.”</li></ul><p>When drafting a new RFC, contributors are asked to include:</p><ul><li>Summary (one paragraph)</li><li>Motivation</li><li>“Guide-level explanation”: features, API input/output, example use-cases</li><li>“Reference-level explanation”: technical implementation details</li><li>Drawbacks: “why should we not do this?”</li><li>Rationale and alternatives
<ul><li>Why is this design the best in the space of possible designs?</li><li>What other designs have been considered? What is the rationale for not choosing them?</li><li>What is the impact of not doing this?</li></ul></li><li>Prior art</li><li>Unresolved questions</li></ul><p>An RFC process front-loads the collaborative work of major changes or additions to an open-source project. That is, instead of waiting until a pull request arrives with lines of code changes, discussion begins early. Contributors are required to put in effort up-front to help others in the project community understand their proposed changes. The project’s TSC and its extended developer community are invited to provide input within the structure of the RFC.</p><p>RFCs do introduce more overhead to a project. A developer can no longer just assume a code contribution speaks for itself; they do also have to put their ideas into words and arguments. However, the RFC process is often a net gain for open-source projects of a certain size and complexity. Any individual or organization is invited to contribute to the project’s growth — they are just asked to work within a structure.</p><h2 id="feature-flags">Feature flags</h2><p>Once an RFC has been “green lighted” by a TSC (or whatever approval critiera is used on a particular project), there are the questions of when and how to merge the related code changes and additions into the project. The sooner code is merged, the easier it will be for others to test and integrate with their own work-in-progress. The later code is merged, the more polished it may be. However, there is also the unfortunate reality of open-source development: The sooner code is merged, the more likely it will introduce bugs and needlessly expand future “surface area” requiring maintenance. The later code is merged, the more likely the developer will disappear, having lost interest in seeing their contribution reach fruition.</p><p>One practice that strikes a balance between “too soon” and “too late” is to use feature flags. New additions are merged into a code base before they are ready for general use. Users can only access one of these new features by setting a configuration “flag” to opt-in. Feature flags provide a means to stage the introduction of major changes and additions. Code changes are merged early and often; functionality changes are hidden from most users until later.</p><p>The feature-flag practice is also used within many web sites and applications. Facebook selectively rolls out feature changes to certain numbers and types of users, much like how junk food manufacturers test their new sodas and chips in small but representative markets.</p><p>The <a href="https://guides.emberjs.com/v3.2.0/configuring-ember/feature-flags/">EmberJS project uses feature flags</a> together with an RFC process. Each RFC typically turns into code hidden behind a feature flag. Advanced users can opt-in to the feature in early releases. After further testing and improvement, the feature flag will be removed and all users will use that code in future releases. (Later, we’ll also discuss <a href="https://www.interline.io/blog/open-source-best-practices-for-otp-summit/#release-management">release management</a>, which helps pace this process of introducing new functionality to a wider audience.)</p><h2 id="contrib-vs-core">“Contrib” vs. core</h2><p>RFCs and feature flags work for contributions that are relevant to all users of a project, but what if a contribution is only relevant to some? An alternative approach is to divide an open-source project into two zones: “core” and “contrib” (that is, contributed). The standards for acceptance for changes to “core” are high — everyone depends on the stability and performance of these components. The standards for “contrib” are lower — these components may offer expanded functionality, but they come with fewer promises of current performance and future maintenance. “Contrib” is an area for experimentation, while “core” is more carefully managed through RFCs and similar types of structure.</p><p>Here’s how the Drupal CMS differentiates between its core and contributed modules:</p><figure><img src="https://www.interline.io/content/images/2024/11/drupal-contrib-modules.png" alt="screenshot of Drupal documentation" width="539" height="394"></figure><p>Note how Drupal also refers to custom modules. In the case of OTP, the reason for our OpenTripPlanner Summit last month is that many potential code contributions to OTP have been accumulating in branches and forks, but few are being merged back into the main OTP repository’s master branch. In effect, these branches and forks are all custom modules — without a clear path to entry to a “contrib” or a “core.”</p><p>To succeed, the core/contrib split does require certain conditions. First, the open-source project must be modular and expandable. That is, “contrib” code should be able to extend or modify aspects of “core” code. Otherwise, all changes would need to happen directly within “core.” (This is a challenge for many projects, OTP included.) Second, “contrib” is <a href="https://blog.startifact.com/posts/against-contrib.html">a poor shorthand</a> for “code developed by non-core committers.” The core/contrib split is best used as a dividing line between the level of commitment a project has to certain functionality, rather than a dividing line between classes of developers.</p><h2 id="automated-acceptance-test-and-performance-suites">Automated acceptance test and performance suites</h2><p>Whether code is going behind a feature flag, into “core”, or into “contrib,” there should be a process for verifying its quality before it is merged. This is important in any type of software development effort; it’s especially important for open-source development, where contributions may be coming from developers of various skill levels, using various tools and development environments.</p><p>Many open-source projects require that new code contributions also come with automated tests. These tests use sample data and conditions to demonstrate that new functionality behaves as expected. Automated tests help prevent regressions (a future change breaking current functionality). Tests also help document functionality, for other developers to understand a system’s goals.</p><p>There are many types of software tests: <a href="https://en.wikipedia.org/wiki/Unit_testing">unit tests</a>, <a href="https://en.wikipedia.org/wiki/Integration_testing">integration tests</a>, <a href="https://en.wikipedia.org/wiki/Smoke_testing_(software)">smoke tests</a>, <a href="https://en.wikipedia.org/wiki/Graphical_user_interface_testing">GUI testing</a>, <a href="https://en.wikipedia.org/wiki/Software_performance_testing#Testing_types">performance testing</a>, security vulnerability scanning, and <a href="https://www.w3.org/wiki/Accessibility_testing">accessibility testing</a>. The specific test types will depend on the type of project. Different mixes of testing approaches are useful for command-line interface scripts, web applications, mobile applications, desktop GUIs, or embedded system modules.</p><p>An open-source project should aim to run automated tests for two overall goals: acceptance and performance. An acceptance test suite tells developers that a proposed code change works as expected and doesn’t break existing functionality. A performance test suite warns developers when proposed code changes will have dramatic changes on speed or resource consumption. For some systems, a single set of tests can inform both acceptance and performance. For other systems, acceptance and performance are more easily tested as separate steps, before and after the piece of software is built or compiled.</p><p>Acceptance and performance tests can be automatically run against every proposed code change by a build/CI/CD server. (CI stands for continuous integration; CD stands for continuous delivery.) We especially like <a href="https://circleci.com/">CircleCI</a> at Interline, and will be blogging more about this soon.</p><p>Acceptance and performance tests often produce a green ✅ or a red ❌ — that is, they give a “yes” or “no” answer to the question of whether a proposed code change is good to merge. However, good test suites also produce more detailed information for ongoing use by a TSC and the regular contributors to an open-source project.</p><p>Here’s a code coverage report for the Transitland Datastore, the web application at the center of the <a href="https://www.transit.land/">Transitland</a> open public-transit data platform. This report is generated by the acceptance test suite. We occasionally review it to ensure that acceptance tests are covering a growing portion of the code bases:</p><figure><img src="https://www.interline.io/content/images/2024/11/transitland-test-coverage.png" alt="screenshot of Transitland Datastore test code coverage" width="1053" height="853"></figure><p>For programming languages and other heavily used software infrastructure, small differences can make major unexpected performance differences. The <a href="https://rubybench.org/">RubyBench</a> project regularly evaluates hundred of actions in the Ruby scripting language and its most important packages. The same tests are run against new releases, enabling comparison with past versions. Developers can use the website to, hopefully, see performance increasing over time:</p><figure><img src="https://www.interline.io/content/images/2024/11/rubybench.png" alt="screenshot of Ruby on Rails benchmark reports over many versions" width="984" height="739"></figure><p>All software projects benefit from testing infrastructure. Testing infrastructure that is easy to access and use is all the more important for open-source software. Learning to read and write tests is an important step in the education of a software engineer. For some, an open-source project will be the first time they encounter auomated test infrastructure. Even for experienced developers, seeing their proposed change produce the red ❌ of a failing change may be the first occasion they have to dig deeply into an unfamiliar code base.</p><p>Writing and running tests does not need to feel like flossing, a boring but necessary chore. When test suites are well structured and made easily accessible, testing helps improve everyone’s confidence in an open-source project and to reduce the risk of contributing.</p><h2 id="automated-dependency-checks">Automated dependency checks</h2><p>Most pieces of open-source software are built upon others. These are software dependencies. Sometimes, dependencies are just copies of code. Hopefully, dependencies are included using a package manager. Package managers make it simple to upgrade dependencies, when new versions are released.</p><p>Just as open-source projects benefit from making automated test suites accessible to developers, projects can benefit from automating and making accessible the process of checking for new dependencies.</p><p>Here’s an email I received a while back from a service called Gemnasium, reporting that a new version of a dependency is available and fixes a known security vulnerability:</p><figure><img src="https://www.interline.io/content/images/2024/11/gemnasium-update.png" alt="screenshot of a dependency security alert" width="672" height="654"></figure><p><a href="https://docs.gitlab.com/ee/user/project/import/gemnasium.html">Gemnasium has now been acquired by Gitlab</a>, where they are beginning to offer a similar automated service. GitHub now also offers its own dependency updates. Some services, like <a href="https://greenkeeper.io/">Greenkeeper</a>, will even perform updates automatically, without waiting for a developer to do so on their own computer.</p><p>Out-of-date dependencies are one of the main ways that software projects open themselves to security vulnerabilities. Closely monitoring dependencies to know when they need to be updated is an important but often forgotten task for open-source projects. Having an automated means of doing so can help turn this from a difficult chore to an easily shared test (similar to automated test suites).</p><p>However, automated dependency checks do come with wrinkles: First, updates about dependencies with security vulnerabilities need to be acted upon quickly. Otherwise, a security alert is a glaring sign saying “come take advantage of this software!” Second, the process of updating dependencies relies on good release habits within those projects. How do you know what has changed within a new version of a dependency? How do you know if any of the changes will require your own project to change? Let’s consider release management next, since it’s through release management that open-source projects can provide stability to build atop each other.</p><h2 id="release-management">Release management</h2><p>The goal of release management is to produce a new version of a software package, ready for use. For commercial software, release management can be as grand an effort as the launch of a Hollywood movie. For open-source software, release management can often be an afterthought. As with all these other practices, having a structured and consistent approach to release management can be a great benefit to an open-source project.</p><h3 id="release-managers">Release manager(s)</h3><p>Who performs the work of producing a new release? For every open-source project that has grown beyond a single core contributor, it’s worth naming a release manager or a release team. This person or people will deserve credit and thanks when the work of producing new releases goes well — and when producing a new release is a challenge, the release manager/team will need sufficient authority to make decisions to resolve problems.</p><p>On the Kubernetes project, there is a “special interest group” (or SIG) focused on release management. <a href="https://github.com/kubernetes/sig-release/blob/master/release-team/README.md">Their responsibilities</a> include:</p><ul><li>Authority to build and publish releases at the published release date under the auspices of the CNCF</li><li>Authority to accept or reject cherrypick merge requests to the release branch</li><li>Authority to accept or reject PRs to the kubernetes/kubernetes master branch during code slush period</li><li>Changing the release timeline as needed if keeping it would materially impact the stability or quality of the release these responsibilities will continue to be discharged by SIG release through the Release Team. This charter grants SIG Release the following additional responsibilities</li><li>Authority to revert code changes which imperil the ability to produce a release by the communicated date or otherwise negatively impact the quality of the release (e.g. insufficient testing, lack of documentation)</li><li>Authority to guard code changes behind a feature flag which do not meet criteria for release</li><li>Authority to close the submit queue to any changes which do not target the release as part of enforcing the code slush period</li></ul><p>Note how the Kubernetes release team uses <a href="https://www.interline.io/blog/open-source-best-practices-for-otp-summit/#feature-flags">feature flags</a> and other tools we’ve discussed previously: They act in a reactive manner, doing whatever is necessary to ship a release. This is in contrast with our previous discussion of <a href="https://www.interline.io/blog/open-source-best-practices-for-otp-summit/#technical-steering-committees">technical steering committees (TSCs)</a> and how they can use tools like feature flags in a proactive manner to pace development. Ideally, an open-source project is able to have both proactive planning (to build up code and features that are worthy of release) and reactive release management (to ensure that releases do happen consistently and regularly).</p><h3 id="release-checklist">Release checklist</h3><p>A release manager/team should have a checklist that they follow each time they “cut” a release. In the short term, this checklist helps make the process of producing a new release easier to repeat. In the long term, this checklist makes it simpler to add new members to a release team and let burnt-out members depart, without losing the “secret” steps of how to produce a release.</p><h3 id="release-notes">Release notes</h3><p>Once a release has been produced, how will users understand what’s included and what changed? Ideally there is a release notes document that includes this information. It can be as simple as a bulleted list (also known as a change-log) or as thorough as an illustrated guide.</p><p>Each month, the Visual Studio Code project cuts a new release and includes illustrated release notes, which are also posted to <a href="https://code.visualstudio.com/updates/v1_25">their blog</a>.</p><p>For the Transitland Datastore, the web application at the center of the <a href="https://www.transit.land/">Transitland</a> open public-transit data platform, we use <a href="https://github.com/github-changelog-generator/github-changelog-generator">a semi-automated process</a> to generate <a href="https://github.com/transitland/transitland-datastore/blob/master/CHANGELOG.md">a change-log from GitHub issues and pull requests</a>.</p><p>The level of detail in release notes or change-log depends in large part on the audience. Visual Studio Code’s release notes are prepared for both end-users and plug-in developers, all of whom need detail. Transitland Datastore’s change-log is prepared for other developers, who are at least somewhat familiar with the existing project and its APIs.</p><h3 id="semantic-versioning">Semantic versioning</h3><p>Typically, each software release gets a version number. The number typically increases with each release. However, there are different ways of formatting and increasing version numbers. This can serve as a signal to users of what and how much changed within the software.</p><p><a href="https://semver.org/">Semantic Versioning</a> (a.k.a. SemVer) is one approach to version numbers. Here’s an overview:</p><p>Given a version number MAJOR.MINOR.PATCH, increment the:</p><ol><li>MAJOR version when you make incompatible API changes,</li><li>MINOR version when you add functionality in a backwards-compatible manner, and</li><li>PATCH version when you make backwards-compatible bug fixes.</li></ol><p>Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.</p><p>This signals to users that, for example, upgrading from 2.4.1 to 2.5.0 won’t require them to make any changes to their own systems. However, upgrading from 2.4.1 to 3.0.0 should only be done after reading release notes and testing the new software; a function they depend upon may have changed.</p><p>Semantic Versioning is a useful ingredient for <a href="https://www.interline.io/blog/open-source-best-practices-for-otp-summit/#automated-dependency-checks">automated dependency checks</a>. If you trust that your dependencies are faithfully following Semantic Versioning principles, you can automatically upgrade to new releases that just increment the “minor” or “patch” version number. Ideally, your project is also using <a href="https://www.interline.io/blog/open-source-best-practices-for-otp-summit/#automated-acceptance-test-and-performance-suites">automated acceptance tests</a>, in order to catch any unexpected side effects that arise from upgrading a dependency.</p><h3 id="lts-long-term-support-versioning">LTS (long-term support) versioning</h3><p>Even if there are advantages to users in upgrading software as soon as possible, there are also costs. Some users will never be able to keep up with the latest version of a project, and may still have support needs and questions about their outdated versions.</p><p>One way out of such situations is to label certain releases as having long-term support (or LTS). LTS releases will be maintained and support for a longer time than other releases. Bug fixes and security fixes made in future releases may also be “backported” so that they are also available to users of an LTS release.</p><p>The Ubuntu operating system project <a href="https://wiki.ubuntu.com/LTS">releases</a> new versions every 9 months. Every 2 years, one of these releases is marked as an LTS release. This LTS release is supported for 5 years.</p><h3 id="train-release-cycles">“Train” release cycles</h3><p>What should actually go into a release? That question is quite dependent on project specifics. Some projects actually try to avoid answering the question. Instead they set up a “train” release cycle, where releases happen on a consistent schedule. Whatever functionality is ready to release when the next date arrives is included in that release.</p><p>Here’s an illustration of how the EmberJS framework releases beta and stable versions every 6 weeks:</p><figure><img src="https://www.interline.io/content/images/2024/11/ember-release-cycle.png" alt="Ember.js release cycle website screenshot" width="796" height="499"></figure><h2 id="codes-of-conduct">Codes of conduct</h2><p>Our discussion of <a href="https://www.interline.io/blog/open-source-best-practices-for-otp-summit/#release-management">release management</a> hinted at some of the problems that can arise in open-source projects. For example: disagreement over features that make it into a release, or unexpected problems merging together code from multiple contributors. Our discussion of <a href="https://www.interline.io/blog/open-source-best-practices-for-otp-summit/#technical-steering-committees">technical steering committees</a> also hinted at the challenges of many people working together on a complex technical project. In the short term, tempers can flare. In the long term, projects can become exclusive clubs, inadvertently sending signals to potential contributors that they are not welcome.</p><p>These days, every open-source project should seriously consider adopting a code of conduct. It’s a proactive means of welcoming new contributors, and it’s a means of providing some “guardrails” for existing contributors to keep in mind.</p><p>The following codes of conduct are widely used and referenced:</p><ul><li><a href="https://www.contributor-covenant.org/">Contributor Covenant</a></li><li><a href="http://citizencodeofconduct.org/">Citizen Code of Conduct</a></li><li><a href="http://geekfeminism.wikia.com/wiki/Community_anti-harassment/Policy">Geek Feminism community anti-harassment policy</a></li></ul><h2 id="the-roadmap">“The roadmap”</h2><p>So far we’ve avoided mentioning the “roadmap.” A roadmap is a forward-looking plan, ordering the development of new functionality and assigning dates to future releases. Developers often find the process of writing roadmaps to be fun and inspiring — the focus is on the potential of tomorrow, rather than the bug fixes or the maintenance tasks of today. Users often find that reading roadmaps provides clarity and confidence — the roadmap tells them what further functionality they can expect, if they invest in committing to use a software package. However, a roadmap does not necessarily represent reality. Projects that are regularly unable to meet their self-imposed targets can lose whatever inspiration and clarity were generated by publishing a roadmap.</p><p>By committing to a firm roadmap, open-source projects can also give up some of their room for flexibility that can be a useful advantage. Open-source projects are often more engineering-driven than proprietary projects. That is, developers and designers who are “closer to the code” are responsible for more decision making. This may provide more opportunities for unexpected optimizations and creativity during implementation. Roadmaps that are overly detailed can hinder opportunistic creation of new functionality and refactoring.</p><p>In reaction to these shortcomings, some open-source projects never write roadmaps. For example, the ReactJS framework:</p><figure><img src="https://www.interline.io/content/images/2024/11/react-roadmap.png" alt="screenshot of a ReactJS issue thread discussing the lack of a project roadmap" width="811" height="1013"></figure><p>Some projects use <a href="https://www.interline.io/blog/open-source-best-practices-for-otp-summit/#train-release-cycles">“train” release cycles</a> to effectively replace roadmaps. For example, on the Django project, whatever additions are completed by a certain date are considered part of the “roadmap” for a release:</p><figure><img src="https://www.interline.io/content/images/2024/11/django-2-1-roadmap.png" alt="screenshot of the Django 2.1 roadmap webpage" width="913" height="278"></figure><p>Finally, some projects find a mid-point. They may use a roadmap to define major themes for future releases, but avoid providing fine details about implementation and functionality that will likely be subject to change. They may use a roadmap to order development priorities and releases, but avoid attaching specific dates. They may use a roadmap to help divide duties among multiple organizations for release management and other maintenance tasks, but avoid listing any further goals for each release.</p><p>When writing roadmaps, it’s too easy to be overconfident in the future of an open-source project. Before committing to a roadmap, consider if some of the other practices we’ve discussed could help achieve similar goals, with more room for flexibility along the way. After considering other practices, if there are still questions that remain that would be best answered by a roadmap-style plan, start with your roadmap only providing a “skeleton” for the most important aspects of the project to plan out in advance.</p><h2 id="the-eternal-rewrite">“The eternal rewrite”</h2><p>We’ve discussed many positive practices of open-source projects. Let’s end by considering some unfortunate outcomes.</p><p>First, consider how many open-source projects encounter problems during major version transitions. A 2.0 release promises to fix all current problems and provide great improvements. But 2.0 still hasn’t arrived, so users continue to work with and improve 1.0 in the meantime. Bug reports keep arriving on 1.0, distracting developers from completing 2.0. It’s unclear what version new users should use, and there are no clear ways for new contributors to help.</p><p>Unfortunately this is what’s happened to the ActiveModelSerializers project, with parallel 0.8, 0.9, and 0.10 versions in various states of usage and completion:</p><figure><img src="https://www.interline.io/content/images/2024/11/ams-version-development.png" alt="screenshot of the ActiveModelSerializers project readme" width="921" height="646"></figure><p>Consider “the eternal rewrite” situation. What of the practices that we discussed could help a project avoid this fate? What of the practices could help a project like this recover and move forward? (As they say in math textbooks, we’ll leave this as an exercise to the reader.)</p><h2 id="the-big-switchover">“The big switchover”</h2><p>Another unfortunate outcome to consider for a version change: “the big switchover.” That is, the new version is available, but major changes leave a large portion of the user community behind.</p><p>For example, this is effectively a decade-long process for users and developers to switch from Python 2.x to 3.x:</p><figure><img src="https://www.interline.io/content/images/2024/11/python-2-to-3.png" alt="screenshot of a report on Python 2 and Python 3 usage levels over time" width="805" height="870"></figure><p>For the Angular project, 1.x and 2.x were so different that some users and developers have never made the leap and continue on 1.x, as if it were a completely different project:</p><figure><img src="https://www.interline.io/content/images/2024/11/angular-1-to-2-1.png" alt="Angularjs 1.0 and Angular 2 logos" width="1024" height="576"></figure><p>“The big switchover” is a good problem to have. It means that a project is so widely used that it has entrenched users. Then again, it may also show a lack of applying some of the open-source practices that we discussed previously. How could a project have better engaged its users, its contributors, and its organizers in the process of building and migrating to a new version?</p><h2 id="a-holistic-view-of-open-source">A holistic view of open-source</h2><p>For a blog post about open-source software, we’ve talked surprisingly little about software engineering or about the business of software. That’s by design. There’s lots to read online about software engineering (The users of <a href="https://stackoverflow.com/">Stackoverflow</a> will try to answer most any question you have). There’s also lots to read about the business of open-source software (even if it’s grand-standing when describing successes and similarly overconfident when judging failures).</p><p>What is often missing are recipes for how to organize and scale open-source projects that have had enough success to get into trouble. That is, projects like OpenTripPlanner that have many users and interested contributors, but a lack of structure to take full advantage of that attention and channel it toward sustainable growth and maintenance. Software does still need to get written, and individuals and organizations do need to earn their income — however, those are necessary but not sufficient ingredients for project success.</p><p>At Interline, we depend upon many open-source projects to succeed. Our clients do, too. That’s why we take this holistic view when evaluating, advising, and contributing to open-source projects. We think it helps improve the odds of success for an open-source project, beyond just code and cash.</p>]]></content:encoded></item><item><title><![CDATA[Producing 200 OpenStreetMap extracts in 35 minutes using a scalable data workflow]]></title><description><![CDATA[Interline now offers OSM Extracts, a service enabling software developers and GIS professionals to download chunks of OpenStreetMap data for 200 major cities and regions around the world.]]></description><link>https://www.interline.io/blog/scaling-openstreetmap-data-workflows/</link><guid isPermaLink="false">67255665b62841000136edcb</guid><dc:creator><![CDATA[Drew Dara-Abrams]]></dc:creator><pubDate>Fri, 25 May 2018 07:00:00 GMT</pubDate><media:content url="https://www.interline.io/content/images/2024/11/argo-workflow-screenshot.png" medium="image"/><content:encoded><![CDATA[<img src="https://www.interline.io/content/images/2024/11/argo-workflow-screenshot.png" alt="Producing 200 OpenStreetMap extracts in 35 minutes using a scalable data workflow"><p>Interline now offers <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a>, a service enabling software developers and GIS professionals to download chunks of OpenStreetMap data for 200 major cities and regions around the world.</p><p>OSM Extracts is simple. It’s a 3 step process that you can run yourself. We’ll describe these steps, as well as how we run those 3 steps for all 200 extracts in parallel on Interline’s cloud infrastructure.</p><figure><img src="https://www.interline.io/content/images/2024/11/osm-extracts-screenshot.png" alt="screenshot of OSM Extracts website" width="1177" height="607"><figcaption><p>A screenshot of <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts by Interline</a>.</p></figcaption></figure><h2 id="creating-osm-extracts-in-3-steps">Creating OSM Extracts in 3 steps</h2><p>Every day at Interline our servers download the latest updates from <a href="https://www.openstreetmap.org/">OpenStreetMap</a>, update our local copies of the planet file, and process this data to generate a variety of geospatial data products. Reliably updating a planet file requires a bit of legwork behind the scenes, which we have encapsulated as helper scripts in our <a href="https://github.com/interline-io/planetutils">PlanetUtils</a> library. This simplifies the process down to 3 steps:</p><ol><li>Run <code>osm_planet_update</code> to download the most recent OSM planet file (released weekly at <a href="https://planet.openstreetmap.org/">planet.openstreetmap.org</a> and other mirrors) and then apply minutely “diffs” to update the planet file to the current point in time.</li><li>Fetch a copy of <a href="https://github.com/interline-io/osm-extracts/blob/master/cities.json">cities.json</a> file from GitHub. This GeoJSON file contains polygons defining the boundaries of each extract, and is a continuation of the extract definitions created by over 100 contributors for the Mapzen Metro Extracts service. The file is open to additions and revisions by all.</li><li>Run <code>osm_planet_extract</code>, specifying the local copy of the cities.json file, to generate extracts for each of the cities/regions.</li></ol><p>Behind the scenes, PlanetUtils handles calls to <code>osmosis</code> and <code>osmconvert</code> to update the planet and generate the extracts. For more information on each of these commands, see the <a href="https://github.com/interline-io/planetutils/blob/master/README.md">PlanetUtils readme</a>, or install the package yourself using Homebrew or Docker.</p><p>This workflow is simple but slow. Updating the planet and generating all 200 extracts on a single machine can take more than a day. In addition to the extracts, the updated planet is the starting point for additional data products, such as our <a href="https://www.interline.io/valhalla/tilepacks/">Valhalla Tilepacks</a>. Valhalla Tilepacks combine the OSM planet file with 1.6Tb of elevation data to produce downloadable data for organizations to run their own instances of the <a href="https://www.interline.io/valhalla/">Valhalla routing engine</a>. All of this heavy lifting requires a substantial mix of computing resources. Parallel workflows to the rescue!</p><h2 id="managing-our-osm-extracts-workflow">Managing our OSM Extracts workflow</h2><figure><img src="https://www.interline.io/content/images/2024/11/argo-workflow-screenshot-1.png" alt="screenshot of Argo workflow dashboard" width="1378" height="1462"><figcaption><p>OSM Extracts and Valhalla Tilepacks are all generated using an Argo workflow. This is screenshot from Argo's management dashboard. To simplify the image, only 2 extract jobs are shown.</p></figcaption></figure><p>At Interline, we have a great appreciation for <a href="https://kubernetes.io/">Kubernetes</a> which powers much of our infrastructure. Kubernetes is ideal for running highly available services and managing complex resources. We wanted our data workflow system to leverage this infrastructure and introduce as little additional complexity as possible. With these constraints, we found that <a href="https://applatix.com/open-source/argo/">Argo Workflow Manager</a> was a great fit for our needs:</p><ul><li>Each Argo workflow step is a <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">Kubernetes pod</a> and <a href="https://www.docker.com/what-container">Docker container</a>, simplifying resource requests and access to secrets (e.g., passwords and access keys for external services).</li><li>Workflows can be described as a directed graph, with an easy syntax for specifying dependencies between steps.</li><li>Argo provides a flexible method for specifying output files and parameters that can be fed into subsequent steps.</li><li>Parallel workflows are well supported, and one step can “fan out” to a number of parallel tasks (as you can see in the diagram above).</li></ul><p>Our Argo workflow resembles the simple 3-step process with a few key adjustments:</p><ol><li>Update our local planet using <code>osm_planet_update</code> and copy to our cloud storage bucket.</li><li>Fetch the <code>cities.json</code> file, and divide the GeoJSON features into smaller chunks of about 8 extracts each.</li><li>Run each chunk as a parallel task to quickly generate all 200 extracts.</li></ol><p>Because each Argo task is defined as a Kubernetes pod, we have very fine-grained control over the resources allocated to each task. The Kubernetes scheduler is very effective at auto-scaling your cluster based, quickly increasing cluster size to run all tasks in parallel, and just as importantly, reducing the cluster size when the tasks are complete. Additionally, because some tasks are CPU intensive and others are memory intensive, we define multiple node pools to ensure a high utilization of cluster resources.</p><p>Currently, we run about 25 tasks of 8 extracts each. With each task requesting an 8 CPU node, generating all 200 extracts in parallel takes about 35 minutes. This workflow also allows us to run multiple data pipelines together, sharing common outputs and reducing the amount of work that has to be duplicated. For instance, the updated planet file generated in the first step above is also used as the input to our Valhalla workflow.</p><h2 id="next-steps-in-the-workflow">Next steps in the workflow</h2><p>Our strategy for working with OpenStreetMap and other data sources is “snout to tail” — that is, we look for opportunities to turn each piece of a giant data download into a useful “meal” for users. We’re working on additional inputs and outputs to Interline’s workflow. Tuning the workflow is also always a work in process. Each week, we find more ways to improve the performance of a step. We look forward to sharing more updates on our workflow in the future.</p><p>In the meantime, we’d like to invite you to add a few next steps to your “professional workflow”:</p><ul><li>If you work with OpenStreetMap data, sign up for the free <a href="https://app.interline.io/osm_extracts/interactive_view">OSM Extracts</a> developer preview.</li><li>If your organization needs to deploy a worldwide routing engine on premise or its in own cloud account, evaluate <a href="https://www.interline.io/valhalla/tilepacks/">Valhalla Tilepacks</a>.</li><li>If your organization runs its own custom data workflow and needs help consider these or other workflow approaches, contact us to discuss Interline’s <a href="https://www.interline.io/consulting/">consulting service</a>. We’re glad to share our experiences with Argo, Kubernetes, and related technologies.</li></ul>]]></content:encoded></item></channel></rss>