Designing our new REST API

Considerations, design & launch of our new API

To API or not to API … that is often the question. And when we need to API, how to design it, what’s important, and critically, what will be easiest for our 3rd party developers?

These are the questions we’re here to answer, to share recent experiences and considerations in designing a new API, all while trying to adhere to conventions, standards, and best practices.

API Priorities

For our new API, we first ponder our priorities. Obviously, we need the API to work, to match our core objects and the required methods, but what about beyond that?

Lots of APIs are just that, basic representations of the system’s core objects and operations, with basic data transfer objects, and that’s it.

But we really want to think beyond that, and focus on the non-core aspects of the API, i.e. the good and smart structure of the API itself, and especially how to make it easier to use, parse, and manage.

So our priorities are:

• Ease of Use — How to make it easy for our 3rd party API users to use it, as API designers often build things for themselves, not the ‘client’.
• Easy to Parse — Part of ‘Ease of Use’, which at its core includes making the JSON results easy to parse and code on the client’s end.
• Consistency — Another part of ease of use is being absurdly consistent, in format, methods, messages, and so on.
• Clear Error Handling — APIs rarely handle and report errors well, so having clear, structured error handling is key to a good API.

API Requests

With the above high-level priorities in mind, what follows are the key design elements we settled on, including why we chose them over alternatives.

Standards

First, we have to make sure we follow basic standards and conventions as best we can, as this just makes common sense. This means doing things right, even if that’s not the easiest to implement.

Second, there are no perfect, or even agreed upon, standards for REST APIs. Yes, most people can describe them vaguely, but it’s not so clear down at the implementation level, e.g. for specific methods, data structures, and error handling. There is a lot of reading between the lines, following common sense, and trying to make it easy for all involved.

Methods

The first area this crops up is with HTTP Methods, such as GET and POST. Many REST APIs limit themself to these two methods and that’s it. But that’s not really right.

In particular, the subtle difference between POST, PUT, and PATCH is important to try to get right. POST creates a new resource, period. It never, ever updates an existing one. PUT and PATCH, on the other hand, are used to update an existing resource, but almost never create one (PUT can create one if the ID is known in advance, and is idempotent).

The difference is PUT replaces (or creates) an existing object, while PATCH updates only some of the fields. This gets complicated in use, so many APIs use PUT for both full and partial updates, but we try to do it right and use both, and make sure it’s documented.

Nouns

Another area that gets glossed over is the nouns, or objects in REST, such as users, cars, orders, etc. These should be plural in all cases, so use ‘orders’ instead of ‘order’, and thus to get a specific order, it is:

GET /orders/123456

Actions

Basic REST says use the HTTP Verbs for everything, such as POST /users to create a new user, etc. But how to do an action on a given user, such as resend an email notification? This can be controversial, but we settled on appending an action to a resource, via a POST such as:

POST /users/{userId}/resend-invite

This is likely a write operation, so we can’t use GET, and we’re not directly modifying the resource, so PUT & PATCH seem wrong, so we end up defaulting to POST.

Parameters

Another common area of confusion is format and casing of parameters. This is because in-line path parameters are usually kebab case (user-id) while JSON fields are traditionally camel case (userId).

This is annoying, as it easily causes errors in coding on both sides, but is still the right format, as it follows convention and standard (and allows for automatic processing by various frameworks).

Kebab for path parameters and camel in JSON.

API Responses

Now that we’ve looked at standardizing the inbound request, we turn to our responses.

Standardization is especially important here, and in this, we are rigorous in that EVERY API call returns the SAME response object structure. This is true whether the response is a simple result, a long list, complex objects, and errors. It’s always the same structure.

This is very important for the client, as it vastly simplifies their response processing, and error handling. Having a fixed structure, even for highly-variable data and responses, makes things much easier to process.

Response Structure

Our response structure is as follows, always returned in this order in a single JSON response:

• timestamp — ISO8601 format timestamp for when the API response was generated. Note this may not exactly match the actual resource creation or modification timestamps.
• success — True or false, where false indicates an error.
• message — A success or error message, human-readable.
• errorCode — Fixed Error Code if success is false, else empty string.
• page — Page of response.
• pageSize — Page size for response.
• totalItems — Total data item count.
• totalPage — Total page count.
• type — Data field object type.
• data — Embedded JSON data object(s), as either a single object or an array.

Only the data field is allowed to be null. All other fields are empty strings “” or 0 when not used. This simplifies and standardizes client processing.

Data Type

The most important result field is the type, which indicates the data type of the object(s) in the data field. Each response contains only a single data type, and all data in the data field will be of that type.

For example, a type might be “UserResponse” which indicates the client should expect the UserResponse object in the data field — this (and all) object types are clearly documented in the API documentation, so it’s very clear what the client should expect and how to process it.

Data Field

The data field itself contains one or more objects of the type specified in the Response’s type field. This will match the documented API response types, e.g. UserResponse.

One complication is how to handle single vs. multiple data objects, and the main question is whether or not to use a JSON array for a single object.

We decided to not use an array, i.e. if this API response type is only ever a single object, we return that object directly, not using an array. Thus this field for a single object is {object} while for multiple objects it’s [{}, {}…].

We do this to simplify the client processing, as the client should not have to deserialize into an array, then find its size, and loop or pull item 0, etc. as this is complicated and can introduce errors for simple API calls.

Instead, documentation for each API endpoint clearly states if the client should expect a single item or an array. For API endpoints that can reutrn multiple objects, we ALWAYS return an array, which may have 0, 1, or more items it — the use of an array is defined by the endpoint, not the data itself. Thus something like GET /user/123/orders may return [], [{}], or [{},{}…].

Options

One option we’re considering is to include another field, such as singleMultiple or array, in the Response object to indicate if the data is an object or array. To date, we have not done this.

Another option is to use the totalItems field to control this, so if totalItems=1, the client can expect an object, and if totalItems>1 it should expect an array. This is messy, though, as it increases the client’s logic and complexity — since most client calls are for specific API endpoints, we’d like them to be able to just use simple code for simple objects, not having to check the size, have a possible array, etc.

A final option is to always use an array, even if there is only one item. This is viable and common, but strikes us as too much work for the client for most simple API calls.

Specifications

We strive to have excellent API documentation, which means a good OpenAPI (i.e. Swagger) specification for the API. These are not that easy to do well, and quite hard to keep up to date, though automated tools can help.

Of course, OpenAPI specs should be done well, including all the right verbs and methods, but especially the schemas, parameters, return codes, and descriptions.

These specs are very helpful, as they drive good documentation (see below), and can be consumed by 3rd party tools, IDEs, etc. to help ensure the clients are making the right calls, the right way.

Be sure to run the resulting YAML file against a YAML checker and against an OpenAPI checker.

Documentation

Even with a good spec file, good API documentation is still required. This includes docs about how to find and access the API, authentication, and overall descriptions of all the standards, objects, etc. mentioned here.

This especially includes the format and meaning of the various object data types and their fields.

Errors Handling

Handling errors well is the hallmark of a great API. There are many different parts of API error handling, some of which often get missed. However, great error handling is extremely helpful to the 3rd party developer calling the API, as they often experience various errors while coding their client, and poor error responses make their lives miserable.

So, just like our non-error responses, we return a standardized error structure and object. This way the client always knows exactly where to look and how to interpret the problem.

Every API response, even non-error ones, have three key fields:

• success — Set true by default, but any error will set this false, so the client knows there was an error, and to look at the Error Code.
• errorCode — Specific fixed Error Code the client can write logic for, such as TOKENEXPIRED.
• errorMessage — Human-readable Error Message, which can change over time (since the client uses the Error Code for logic, never the Error Message). Note this is usually a technical message, rarely suitable to show the end user, but valuable in logs and debug messages.

In addition, some errors result in specific HTTP Response Codes, basically a non-200 code, which carries very specific meaning. Many APIs return only a 200, 400, and 404 when something goes, but these are not really detailed enough for clients to use.

Given the HTTP and REST standards have a much richer set of Status Codes, we should use them to also convey meaning, in addition to the embedded Error Code and Error Message, mentioned above.

Also note some HTTP errors don’t allow us to return any content, either because the error was generated upstream in our backend frameworks (common for 4xx errors), or are not properly formed or authenticated. In these case, returning the right HTTP Status Code is very important, as it’s all the client has to go on for troubleshooting.

Key HTTP Status Codes we use include:

• 400 — for bad formats, bad JSON, bad URL, etc. Often generated by the upstream framework, and hard to send Error Codes or Messages for, but we should try.
• 400 — Bad parameters, such as page=-1 — We must return an Error Code and Message for this.
• 401 — Bad authentication, such as invalid key. Note a 401 means they can't access ANY resources, unlike a 403 which is resource-specific.
• 403 — No access to this object/method on that resource.
• 404 — Path, resource, etc. do not exist. For both path & resource issues, e.g. /userz/ or /users/1234.
• 404 — When a resource exists, but the caller has no access & is not allowed to know about it. This is common for a GET on resources the caller does not own, such as other users’ orders.
• 405 — Bad method on a resource, for resources the caller can know exists (careful with this), e.g. a GET but no POST, etc.
• 406 — Bad MIME type accepted (e.g. client not include Accept: application/json). Often set by the upstream framework.
• 422 — System Error (non-500) — A catchall we use for various client errors, such as missing body, other API structure violations
• 415 — Bad MIME type, usually if Content-Type: application/json and charset=utf-8.
• 201 — Created resource due to POST. Return a Location: header with the URL of new object for a future GET.
• 202 — Accepted request, not completed (maybe async, etc.). Should include how to get status, such as: Location: /api/status/12345
• 204 — When there is no response, such as for a DELETE (no resource to return), or action resource (resend-email). Should still return a response object with success = true. Note any empty response should send 204, such as GET /users?name=NoOneHere
• 303 — Creating an object via an async process. Should include how to get status in Location: header.
• 501 — Not implemented, for functions not yet done. Useful as placeholders and for documented but not finished APIs.

Implementation

All of the above is the how and why we designed the API. Below are some details on how we actually implemented it. We hope they are useful.

Our API is built in Java using Spring Boot, and so we use various Spring conventions for its implementation. Other languages and frameworks will be similar.

Authentication

Our API processes requests for both an individual platform user, using an API Key, and on behalf of a platform user, using OAuth. Thus requests arrive with two types of Authorization: headers and we differentiate by header prefix, using Apikey: or Bearer: prefixes. The Apikey: prefix is non-standard but lets us easily differentiate and validate the header and value.

We extensively validate they incoming key or token, verifying its length, format, charset, characters, JWT options, etc. before even trying to check it — this helps protect our validation libraries (such as JWT’s alg=non issue) from various attacks.

The system then validates the token itself, and if valid, attaches the user (direct for Apikey and on behalf of for OAuth JWT Tokens) to the session so the API has user info in its context when it runs.

This all happens in the Spring framework via interceptors, so it’s before a request is routed or otherwise checked for anything.

Paths & Verbs

In Spring Boot, we specify the path and verb explicitly in our controllers, such as: @GetMapping(“/users/{user-id}”) and the framework will automatically reject invalid paths, verbs, etc. and in some cases, invalid parameters.

Response Objects

All responses return the same object type, in our case, ApiResponse, which has all the fields mentioned above. The class itself sets various defaults, including empty strings “” for all string fields and 0 for numerics. This ensures valid defaults if they are not further set in the main API code.

API Logic & Processing

Our Spring controllers do minimal work and then hand all the requests and parameters off to a service layer, such as UserService or OrderService where the real logic lies.

The service layer actually builds up the response object on its own, simplifying the controller layer and allows directly setting of response data in the real code.

Note some architectures instead use a DTO to set and return data from the service layer, but we prefer a simpler, more direct approach (though this has its own complications).

Once the service layer has the data to return, it builds the data object(s), attaches them to the response object, sets the size and type, and returns.

Error Processing

General error processing is done in the service layer, so if there is a failure or exception, all they have to do is log it, then directly set the success field to false, the errorCode to a defined code, and the message field to something describing the error. Then they return and everything else is taken care of.

Cross-Checking

As mentioned, our controllers don’t have much logic, but they do have a key final step, in calling our cross-checking validator. This important function is used to catch coding and processing errors on our part, as it cross-checks and validates the various parts of the response to make sure it’s valid.

It’s called like this at the end of a controller:

return checkApiResponse( ResponseEntity.status(HttpStatus.OK).body(response));

And it’s checking for the following, logging and returning a separate simplified error response when if finds any of these things (and hoping the client will then call us to report these issues):

• Null response — Should never happen, but might if there is some unhandled exception or internal error.
• Data Type — If the data object is not null, ensure a valid and matching data type is set.
• Data Size — Cross-checks the actual size of any data array and the size indicated in the response.
• Paging — Checking that paging response make sense
• Others — Adding other checks as we find issues

Summary

That is our API, along with how and why we designed it that way. We hope it’s useful and are happy to get feedback, questions, and suggestions.

I’m Steve Mushero, a fractional CTO for early-stage startups. I help CEOs and CTOs build confidence in their product, processes, and people. Learn more at SteveMushero.com and view my profile on LinkedIn.