On the WordPress REST API: Reflecting on the past and looking to the future

Filed Under

Published

April 3, 2015

9 Comments

If you go to a WordCamp and spend anytime around developers, it is hard to not notice the excitement around the new WordPress REST API.

The REST API team just released version 1.2 of the plugin, which they are continuously developing with the goal of merging it into WordPress core.

It’s a big project, much bigger than Ryan McCue—who, along with Rachel Baker, leads the project—ever thought it would be. I recently spoke with Ryan about the challenges of such a huge project and about the complexities in creating a “universal API” for every WordPress site, as well as the plans for the core merge.

Version 1.2 and backwards compatibility

Version 1.2 of the plugin is the last planned major release of the REST API as a plugin. The team is now focused on version 2.

Since the release of version 1, Ryan has been committed to supporting it and ensuring backwards compatibility—something that he and the REST API team are still committed to.

Version 2, the code that will merge into WordPress core, will not be fully backwards compatible with version 1, however, the plugin will continue on as a backwards-compatibility layer. That way anything developed using the plugin version, as long as the plugin is active, will work once the REST API is in core.

“We’re not going in and recreating this from scratch, and we’re certainly not breaking things unless we have to, so porting things over from version 1 to version 2 will be simple,” Ryan said in an interview.

In version 2, all routes will now be prefixed. Core will use a “wp” prefix and all custom routes will have their own prefix. In addition, the “wp-json” prefix currently in use for all routes may go away entirely. Sites using the plugin will of course get the older wp-json routes rerouted to the new routes once they’re in core.

“The idea is that externally, to anyone using the plugin, that our compatibility layer will make it look exactly the same to anyone using version 1.2. So version 1.2 is kind of like a fixed point in that regard,” Ryan said… “But internally, it will actually use version 2 and core infrastructure… so code written for version 1 can automatically benefit from any bug fixes that come out… and still have security maintenance… and we don’t have to support two versions.”

The core merge: possibly in two phases

Ryan hopes to have a usable version of version 2 within “the next month or so” to be released as a beta. The version 2 beta will be used as a merge proposal for core and, ideally, it will be merged into core in 2015.

The REST API codebase is essentially split into two parts: The first is the infrastructure for creating a RESTful API. The other part is the default endpoints it provides.

It is possible that the infrastructure will go into core first and then the endpoints will be added at a later date. If that happens, the infrastructure will be labeled initially as an experimental API. This would mimic how the Heartbeat API was introduced slowly into core, over more than one version.

The universal API

I asked Ryan about the challenge of making a “universal API” to be included in so many sites. He said that the biggest challenge is balancing all of the different concerns. Adding the default endpoints to core has a lot of complexity in terms of ensuring the right data is there, and balancing the varied and sometime competing concerns is a major challenge. This why getting it right has taken longer than expected.

He sees two primary core use cases for the API—both of which have conflicting needs.

The first use case is what he calls “service developers.” An example of this type of user is a news organization wanting to create an app based on their website providing API data as a service to the app. These users are more likely to use custom endpoints. So, for them, the infrastructure is more important than the default endpoints themselves.

A good example is Wired Magazine. Wired recently replaced multiple WordPress installs with one install, linked to multiple front-ends via the REST API. Ryan’s favorite part about this project is that the REST API will allow developers to choose their own way of working with WordPress. No longer will developers have to use a theme in the traditional way. They can use the JSON data anyway they want.

“That’s one of the use cases I really like,” Ryan said. “The fact that someone who doesn’t know WordPress that well can go and start creating frontend websites with this… It expands how many people can use and develop for WordPress.”

The other use case Ryan predicts is WordPress mobile apps. For these cases, the goal is to make the API work “consistently across every site in existence.” This use leads to an impossible expectation of the same data being provided from every site.

“Some of these problems we don’t want to try and handle because there is way too much there,” Ryan said. “The flip-side of all of this is that even if we know the sites have the same data, they may not have the same fields… We are trying to work out the best balance… between the flexibility we need to allow for plugin developers and the strictness that clients need.”

The ever-expanding scope of the REST API

Ryan has been working on the WordPress REST API for a very long time. To give you an idea of exactly how long that is, when he started, he thought it could be included in WordPress 3.6. The longer time frame for this project than he originally anticipated is not just because he underestimated the amount of work, but because the scope of the project has grown so much.

Originally he saw the REST API as ” just a way to get at the underlying data in WordPress and not much more than that.” Some of the code from the first release, which was very minimalistic, and handled a much smaller amount of data, still exists to this day.

The limit on just the four core objects—posts, users, taxonomies, and meta data—was decided early on, once the project grew. That’s limiting, but it sets up the REST API once it is in core to cover options and other data types.

Like all of WordPress, the REST API is designed to be extensible. The infrastructure for handling the rest of the data types, or doing whatever you want as a developer is there.

“The infrastructure of the API itself, supports basically anything you can throw at it. If you take away the core endpoints, it is essentially a framework for building APIs, and you can build those however you like,” said Ryan.

In his mind, the biggest misconception about the REST API project is that people are “going to see it as a magic bullet that solves all of these problems.” For example, generic response handling. A mobile app can’t automatically handle every single response that any WordPress site will be able to send to them. Solving the problem of a generic API hasn’t been solved and the WordPress REST API will not solve that problem.

“We want to allow this flexibility to plugin developers,” Ryan said, “plugin developers also have to understand that with the flexibility they are granted the have to take into mind things like this, and it is tough to convey this, especially for service sites… Trying to convey to people that they can remove endpoints, but they need to be aware that this will break stuff. Adding data to a response will be easier to do, as its not a big problem for clients, but it will be harder to remove data that is expected by a client.”

Forward the foundation (for your own API)

Ryan and more than 50 contributors are tasked with adding a RESTful API to tens of millions of websites. These are sites that they do not know the data structure of, or the needs of. In other words, it’s a huge task.

What I hadn’t understood personally before my conversation with Ryan is that this project isn’t just about creating a perfect API that works for tens of millions of sites. It’s really about adding a standard set of tools for anyone to make their own API, as well as a totally functional, but generic API.

The REST API is shaping up to be an awesome reflection of WordPress itself. WordPress is a tool that provides a basic starting point for a CMS in five minutes or less, which can be expanded on pretty much infinitely.

The REST API is no different. It provides a fully functional API out of the box, ready for anyone to use, with all the power needed for site and plugin developers to easily customize.

There are 9 comments

April 3, 2015 #

modemlooper

“It’s really about adding a standard set of tools for anyone to make their own API, as well as a totally functional, but generic API.”

That’s the best thing.
April 3, 2015 #

Mark Simchock

Help me out here…this is going into core why?

Obviously the REST API is big, to put it lightly. But why core? Don’t get me wrong, I’m sure there are some real benefits. But do those out number the number of sites that might not need, nor want, this functionality?
- April 3, 2015 #
  
  Otto
  
  If you don’t use it, then it doesn’t do anything. No biggie. But you probably will use it, even if you don’t know it. The mobile apps will probably use it. Anything that talks to WordPress sites will probably use it. Some types of themes will use it. Mobile and JS based things are the future. Making WordPress capable of this easily keeps it relevant and expanding.
  - April 3, 2015 #
    
    Mark Simchock
    
    So if i don’t see it then there are no potential security exploits? Nothing can go wrong, just because I can’t see? Never a bug? Really? Oh. Good to know. 🙂 (Note: I’m being a bit over the top but these issues are real.)
    
    At the end of the day it adds still more bulk (and bloat?) where none is 100% necessary. i’m pretty sure that anyone wanting to use it will be able to install an activate a plugin. Ya think? 🙂 Or is the plugin architecture not up to the challenge? Maybe that is where we need core work then? Perhaps?
    
    Finally, as a third-party add-on (read: plugin), it’s more likely to encourage forking / innovation / experimentation, etc. As it is, that’s pretty much going to be crushed and subject to all the rules, regs, M&Ps, etc. that come along with it being buried core.
    
    At a pure code level, you might be right. But there are bigger things at work here than simple X lines of code. I think that until there’s more certainty (read: real data not speculation) about the number / % of live / production installs using it, there’s nothing lost by leaving it as a plugin. Correct? Or is this more of press release and/or core bragging rights thing?
    - April 6, 2015 #
      
      Ryan McCue
      
      I’d like to address some of your points here @ChiefAlchemist:disqus 🙂
      
      > So if i don’t see it then there are no potential security exploits?
      Nothing can go wrong, just because I can’t see? Never a bug? Really? Oh.
      Good to know. 🙂 (Note: I’m being a bit over the top but these issues
      are real.)
      
      All software has bugs, it’s the nature of the beast. What we’re trying to do, however, is to keep those as low as possible. When the API goes into core, it’ll be one of the most unit-tested pieces of code in all of WP. It’s also got a tonne of eyes on it, and will go through quite a few more audits before it goes into core; certainly more than most other patches.
      
      > At the end of the day it adds still more bulk (and bloat?) where none is 100% necessary.
      
      Nothing is 100% necessary, and you can always keep using the current version of WP if you’d like. However, we need to make progress in WP to ensure it’s competitive with other solutions.
      
      In addition, I think you’re missing that this will be part of core, not just tacked on. It’ll be used by the admin itself, along with themes and plugins. Existing code in core to handle administration can actually be removed and replaced with REST API code, which is a benefit to code quality and stability. (Again, the API will be unit tested, the bespoke endpoints for admin data right now are either undertested or not tested at all.)
      
      In fact, maybe some day, we can deprecate some of our less tested/battle-hardened things like XML-RPC, or convert them over to be based on the same infrastructure.
      
      > i’m pretty sure that anyone wanting to use it will be able to install an activate a plugin
      
      I disagree. We need to aim for the 80% use case, and most users aren’t going to understand they need to install an extra plugin to things to work, like a mobile app, or an external (IFTTT/Zapier/etc) integration, or a theme. There are certainly things we can and should improve in the plugin system (dependencies, anyone?), but it’s not a zero-sum game: we can do both.
      
      > Finally, as a third-party add-on (read: plugin), it’s more likely to encourage forking / innovation / experimentation, etc.
      
      I don’t really agree with this either (but with a caveat). You can create plugins for the plugin already, and that won’t change in core. Major experimentation on the plugin itself isn’t really happening, we’re pretty stable. There are things we’re changing in v2, but those could have also happened in a plugin to try them out anyway.
      
      That said, I think we currently have a bit of a nicer contribution path for a lot of developers. I know the dotorg/meta team are working on improving the core process, and I’m looking forward to seeing things from that. Rest assured that as a core contributor and also a lazy developer, I want to make it as streamlined as possible.
      
      > As it is, that’s pretty much going to be crushed and subject to all the
      rules, regs, M&Ps, etc. that come along with it being buried core.
      
      Any rules that apply to core code already apply to the REST API. All of the maintainers and developers on the API are seasoned core contributors, and we understand the concerns of the core team. For example, one of the biggest issues with core contribution is backwards compatibility, but we’re also 100% backwards compatible in v1.
      
      Core isn’t just this monolithic beast that is untouchable by mere mortals. It’s worked on by a bunch of really smart people who have come to the consensus that certain things are a good idea. 🙂
      - April 6, 2015 #
        
        Leo
        
        Awesome stuff @rmccue:disqus 🙂 Would REST API + HHVM/PHP 7 on a WordPress installation with an ecommerce plugin be able to handle large amounts of I/O, especially for popular ecommerce sites?
        
        2) Would having REST API help with making WordPress + WooCommerce, PCI-DSS compliant in anyway? 🙂
      - April 9, 2015 #
        
        Mark Simchock
        
        Ryan McCue thanks for the detailed reply. Great work across the board brother. Thank you for all your efforts.
        
        You said “All software has bugs, it’s the nature of the beast.”
        
        Thanks. My point exactly. Which leads us to today:
        
        https://make.wordpress.org/core/2015/04/09/wp-rest-api-critical-security-release/
        
        Now please explain to the rest of us (no pun intended) given the sophistication and complexity of the REST API code base (i.e., Nacin has gone on record about this, yes?) , why it’s necessary (note: this is the key word) to expose sites not using the API to 100% unnecessary risk?
        
        I hear ya, and I agree. When you get to the 50%, to 60% to 70% rate of usage, that’s a great problem to have. We’d all love to see that. Yes! Start talking about into core then. And yes, at 80% putting it in core makes pure perfect sense. I’ll be as silent as snow then. I promise 🙂
        
        But that’s not where we’re at, are we? So why is the reputation of the brand – let alone the security of the MILLIONS of users of the product. – being put at risk for what is current edge-esque use cases?
        
        That’s all that I’m asking.
        
        Why the rush? What’s to be gained? How is risk for the non-using millions going to be mitigated (if it can given the nature of engineering software)?
        
        Neither you nor @Otto4242:disqus have answered this question *directly*, or did I miss something? I’m just wanting to have someone explain for the punters how the risk of sooner out weighs the lower risk of later. Let’s all keep in mind that given the scale of WP market share, even a small delta is massive, yes?
      - April 9, 2015 #
        
        Mark Simchock
        
        Ryan McCue – I would like to address something directlly.
        
        I said: “Finally, as a third-party add-on (read: plugin), it’s more likely to encourage forking / innovation / experimentation, etc.”
        
        You said: “I don’t really agree with this either (but with a caveat).”
        
        You might not think it doesn’t apply, which I suppose is your caveat, but the truth of the the ideal at 50,000 ft is when you lock something down, you lock something down (i.e., stifle innovation). There’s really nothing to debate here. This is just life 🙂
  - April 3, 2015 #
    
    Mark Simchock
    
    Otto. To be clear, those are all use cases. But that doesn’t make them “evidence” for being in the core product. fwiw, this effectively creates a third product (i.e., WP.com, WP.org, and WP.org + REST API). Does that (potential) confusion help to hurt the brand? Does it add to the less favorable perceptions (and truths?) or does it solve them? That’s is what I’m asking. I understand the use cases.