Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

January 23 2014

19:41

Validation and Exception Handling: From the UI to the Backend

Sooner or later in your programming career you will be faced with the dilemma of validation and exception handling. This was the case with me and my team also. A couple or so years ago we reached a point when we had to take architectural actions to accommodate all the exceptional cases our quite large software project needed to handle. Below is a list of practices we came to value and apply when it comes to validation and exception handling.


Validation vs. Exception Handling

When we started discussing our problem, one thing surfaced very quickly. What is validation and what is exception handling? For example in a user registration form, we have some rules for the password (it must contain both numbers and letters). If the user enters only letters, is that a validation issue or an exception. Should the UI validate that, or just pass it to the backend and catch any exceptions that my be thrown?

We reached a common conclusion that validation refers to rules defined by the system and verified against user provided data. A validation should not care about how the business logic works, or how the system for that matter works. For example, our operating system may expect, without any protests, a password composed of plain letters. However we want to enforce a combination of letters and numbers. This is a case for validation, a rule we want to impose.

On the other hand, exceptions are cases when our system may function in an unpredicted way, wrongly, or not at all if some specific data is provided in a wrong format. For example, in the above example, if the username already exists on the system, it is a case of an exception. Our business logic should be able to throw the appropriate exception and the UI catch and handle it so that the user will see a nice message.


Validating in the User Interface

Now that we made clear what our goals are, let’s see some examples based on the same user registration form idea.

Validating in JavaScript

To most of today’s browsers, JavaScript is second nature. There is almost no webpage without some degree of JavaScript in it. One good practice is to validate some basic things in JavaScript.

Let’s say we have a simple user registration form in index.php, as described below.

<!DOCTYPE html>
<html>
	<head>
		<title>User Registration</title>
		<meta charset="UTF-8">
	</head>
	<body>
		<h3>Register new account</h3>
		<form>
			Username:
			<br/>
			<input type="text" />
			<br/>
			Password:
			<br/>
			<input type="password" />
			<br/>
			Confirm:
			<br/>
			<input type="password" />
			<br/>
			<input type="submit" name="register" value="Register">
		</form>
	</body>
</html>

This will output something similar to the image below:

RegistrationForm

Every such form should validate that the text entered in the two password fields are equal. Obviously this is to ensure the user does not make a mistake when typing in his or her password. With JavaScript, doing the validation is quite simple.

First we need to update a little bit of our HTML code.

<form onsubmit="return validatePasswords(this);">
	Username:
	<br/>
	<input type="text" />
	<br/>
	Password:
	<br/>
	<input type="password" name="password"/>
	<br/>
	Confirm:
	<br/>
	<input type="password" name="confirm"/>
	<br/>
	<input type="submit" name="register" value="Register">
</form>

We added names to the password input fields so we can identify them. Then we specified that on submit the form should return the result of a function called validatePasswords(). This function is the JavaScript we’ll write. Simple scripts like this can be kept in the HTML file, other, more sophisticated ones should go in their own JavaScript files.

<script>
	function validatePasswords(form) {
		if (form.password.value !== form.confirm.value) {
			alert("Passwords do not match");
			return false;
		}
		return true;
	}

</script>

The only thing we do here is to compare the values of the two input fields named “password” and “confirm“. We can reference the form by the parameter we send in when calling the function. We used “this” in the form’s onsubmit attribute, so the form itself is sent to the function.

When the values are the same, true will be returned and the form will be submitted, otherwise an alert message will be shown telling the user the passwords do not match.

PasswordDoNotMatchAlert

HTML5 Validations

While we can use JavaScript to validate most of our inputs, there are cases when we want to go on an easier path. Some degree of input validation is available in HTML5, and most browsers are happy to apply them. Using HTML5 validation is simpler in some cases, though it offers less flexibility.

<head>
	<title>User Registration</title>
	<meta charset="UTF-8">
	<style>
		input {
			width: 200px;
		}
		input:required:valid {
			border-color: mediumspringgreen;
		}
		input:required:invalid {
			border-color: lightcoral;
		}
	</style>
</head>
<body>
	<h3>Register new account</h3>
	<form onsubmit="return validatePasswords(this);">
		Username:
		<br/>
		<input type="text" name="userName" required/>
		<br/>
		Password:
		<br/>
		<input type="password" name="password"/>
		<br/>
		Confirm:
		<br/>
		<input type="password" name="confirm"/>
		<br/>
		Email Address:
		<br/>
		<input type="email" name="email" required placeholder="A Valid Email Address"/>
		<br/>
		Website:
		<br/>
		<input type="url" name="website" required pattern="https?://.+"/>
		<br/>
		<input type="submit" name="register" value="Register">
	</form>
</body>

To demonstrate several validation cases, we extended our form a little bit. We added an email address and a website also. HTML validations were set on three fields.

  • The text input username is just simply required. It will validate with any string longer than zero characters.
  • The email address field is of type “email” and when we specify the “required” attribute, browsers will apply a validation to the field.
  • Finally, the website field is of type “url“. We also specified a “pattern” attribute where you can write your regular expressions that validate the required fields.

To make the user aware of the state of the fields, we also used a little bit of CSS to color the borders of the inputs in red or green, depending on the state of the required validation.

HTMLValidations

The problem with HTML validations is that different browsers behave differently when you try to submit the form. Some browsers will just apply the CSS to inform the users, others will prevent the submission of the form altogether. I recommend you to test your HTML validations thoroughly in different browsers and if needed also provide a JavaScript fallback for those browsers that are not smart enough.


Validating in Models

By now many people know about Robert C. Martin’s clean architecture proposal, in which the MVC framework is only for presentation and not for business logic.

HighLevelDesign

Essentially, your business logic should reside in a separate, well isolated place, organized to reflect the architecture of your application, while the framework’s views and controllers should control the delivery of the content to the user and models could be dropped altogether or, if needed, used only to perform delivery related operations. One such operation is validation. Most frameworks have great validation features. It would be a shame to not put your models at work and do a little validation there.

We will not install several MVC web frameworks to demonstrate how to validate our previous forms, but here are two approximate solutions in Laravel and CakePHP.

Validation in a Laravel Model

Laravel is designed so that you have more access to validation in the Controller where you also have direct access to the input from the user. The built-in validator kind of prefers to be used there. However there are suggestions on the Internet that validating in models is still a good thing to do in Laravel. A complete example and solution by Jeffrey Way can be found on his Github repository.

If you prefer to write your own solution, you could do something similar to the model below.

class UserACL extends Eloquent {
    private $rules = array(
        'userName' => 'required|alpha|min:5',
        'password'  => 'required|min:6',
		'confirm' => 'required|min:6',
		'email' => 'required|email',
		'website' => 'url'
    );

    private $errors;

    public function validate($data) {
        $validator = Validator::make($data, $this->rules);

        if ($validator->fails()) {
            $this->errors = $validator->errors;
            return false;
        }
        return true;
    }

    public function errors() {
        return $this->errors;
    }
}

You can use this from your controller by simply creating the UserACL object and call validate on it. You will probably have the “register” method also on this model, and the register will just delegate the already validated data to your business logic.

Validation in a CakePHP Model

CakePHP promotes validation in models as well. It has extensive validation functionality at model level. Here is about how a validation for our form would look like in CakePHP.

class UserACL extends AppModel {

    public $validate = [
		'userName' => [
			'rule' => ['minLength', 5],
			'required' => true,
			'allowEmpty' => false,
			'on' => 'create',
			'message' => 'User name must be at least 5 characters long.'
		],
        'password' => [
            'rule'    => ['equalsTo', 'confirm'],
            'message' => 'The two passwords do not match. Please re-enter them.'
        ]
    ];

    public function equalsTo($checkedField, $otherField = null) {
		$value = $this->getFieldValue($checkedField);
        return $value === $this->data[$this->name][$otherField];
    }

	private function getFieldValue($fieldName) {
	    return array_values($otherField)[0];
	}
}

We only exemplified the rules partially. It is enough to highlight the power of validation in the model. CakePHP is particularly good at this. It has a great number of built-in validation functions like “minLength” in the example and various ways to provide feedback to the user. Even more, concepts like “required” or “allowEmpty” are not actually validation rules. Cake will look at these when generating your view and put HTML validations also on fields marked with these parameters. However rules are great and can easily be extended by just simply creating methods on the model class as we did to compare the two password fields. Finally, you can always specify the message you want to send to the views in case of validation failure. More on CakePHP validation in the cookbook.

Validation in general at the model level has its advantages. Each framework provides easy access to the input fields and creates the mechanism to notify the user in case of validation failure. No need for try-catch statements or any other sophisticated steps. Validation on the server side also assures that the data gets validated, no matter what. The user can not trick our software any more as with HTML or JavaScript. Of course, each server side validation comes with the cost of a network round-trip and computing power on the provider’s side instead of the client’s side.


Throwing Exceptions from the Business Logic

The final step in checking data before committing it to the system is at the level of our business logic. Information that reaches this part of the system should be sanitized enough to be usable. The business logic should only check for cases that are critical for it. For example, adding a user that already exists is a case when we throw an exception. Checking the length of the user to be at least five characters should not happen at this level. We can safely assume that such limitations were enforced at higher levels.

On the other hand, comparing the two passwords is a matter for discussion. For example, if we just encrypt and save the password near the user in a database, we could drop the check and assume previous layers made sure the passwords are equal. However, if we create a real user on the operating system using an API or a CLI tool that actually requires a username, password, and password confirmation, we may want to take the second entry also and send it to a CLI tool. Let it re-validate if the passwords match and be ready to throw an exception if they do not. This way we modeled our business logic to match how the real operating system behaves.

Throwing Exceptions from PHP

Throwing exceptions from PHP is very easy. Let’s create our user access control class, and demonstrate how to implement a user addition functionality.

class UserControlTest extends PHPUnit_Framework_TestCase {
	function testBehavior() {
		$this->assertTrue(true);
	}
}

I always like to start with something simple that gets me going. Creating a stupid test is a great way to do so. It also forces me to think about what I want to implement. A test named UserControlTest means I thought I will need a UserControl class to implement my method.

require_once __DIR__ . '/../UserControl.php';
class UserControlTest extends PHPUnit_Framework_TestCase {

	/**
	 * @expectedException Exception
	 * @expectedExceptionMessage User can not be empty
	 */
	function testEmptyUsernameWillThrowException() {
		$userControl = new UserControl();
		$userControl->add('');
	}

}

The next test to write is a degenerative case. We will not test for a specific user length, but we want to make sure we do not want to add an empty user. It is sometimes easy to lose the content of a variable from view to business, over all those layers of our application. This code will obviously fail, because we do not have a class yet.

PHP Warning:  require_once([long-path-here]/Test/../UserControl.php):
failed to open stream: No such file or directory in
[long-path-here]/Test/UserControlTest.php on line 2

Let’s create the class and run our tests. Now we have another problem.

PHP Fatal error:  Call to undefined method UserControl::add()

But we can fix that, too, in just a couple of seconds.

class UserControl {

	public function add($username) {

	}

}

Now we can have a nice test failure telling us the whole story of our code.

1) UserControlTest::testEmptyUsernameWillThrowException
Failed asserting that exception of type "Exception" is thrown.

Finally we can do some actual coding.

public function add($username) {
	if(!$username) {
		throw new Exception();
	}
}

That makes the expectation for the exception pass, but without specifying a message the test will still fail.

1) UserControlTest::testEmptyUsernameWillThrowException
Failed asserting that exception message '' contains 'User can not be empty'.

Time to write the Exception’s message

public function add($username) {
	if(!$username) {
		throw new Exception('User can not be empty!');
	}
}

Now, that makes our test pass. As you can observe, PHPUnit verifies that the expected exception message is contained in the actually thrown exception. This is useful because it allows us to dynamically construct messages and only check for the stable part. A common example is when you throw an error with a base text and at the end you specify the reason for that exception. Reasons are usually provided by third party libraries or application.

/**
 * @expectedException Exception
 * @expectedExceptionMessage Cannot add user George
 */
function testWillNotAddAnAlreadyExistingUser() {
	$command = \Mockery::mock('SystemCommand');
	$command->shouldReceive('execute')->once()->with('adduser George')->andReturn(false);
	$command->shouldReceive('getFailureMessage')->once()->andReturn('User already exists on the system.');
	$userControl = new UserControl($command);
	$userControl->add('George');
}

Throwing errors on duplicate users will allow us to explore this message construction a step further. The test above creates a mock which will simulate a system command, it will fail and on request, it will return a nice failure message. We will inject this command to the UserControl class for internal use.

class UserControl {

	private $systemCommand;

	public function __construct(SystemCommand $systemCommand = null) {
		$this->systemCommand = $systemCommand ? : new SystemCommand();
	}

	public function add($username) {
		if (!$username) {
			throw new Exception('User can not be empty!');
		}
	}

}

class SystemCommand {

}

Injecting the a SystemCommand instance was quite easy. We also created a SystemCommand class inside our test just to avoid syntax problems. We won’t implement it. Its scope exceeds this tutorial’s topic. However, we have another test failure message.

1) UserControlTest::testWillNotAddAnAlreadyExistingUser
Failed asserting that exception of type "Exception" is thrown.

Yep. We are not throwing any exceptions. The logic to call the system command and try to add the user is missing.

public function add($username) {
	if (!$username) {
		throw new Exception('User can not be empty!');
	}

	if(!$this->systemCommand->execute(sprintf('adduser %s', $username))) {
		throw new Exception(
				sprintf('Cannot add user %s. Reason: %s',
						$username,
						$this->systemCommand->getFailureMessage()
				)
			);
	}
}

Now, those modifications to the add() method can do the trick. We try to execute our command on the system, no matter what, and if the system says it can not add the user for whatever reason we throw an exception. This exception’s message will be part hard-coded, with the user’s name attached and then the reason from the system command concatenated at the end. As you can see, this code makes our test pass.

Custom Exceptions

Throwing exceptions with different messages is enough in most cases. However, when you have a more complex system you also need to catch these exceptions and take different actions based on them. Analyzing an exception’s message and taking action solely on that can lead to some annoying problems. First, strings are part of the UI, presentation, and they have a volatile nature. Basing logic on ever changing strings will lead to dependency management nightmare. Second, calling a getMessage() method on the caught exception each time is also a strange way to decide what to do next.

With all these in mind, creating our own exceptions is the next logical step to take.

/**
 * @expectedException ExceptionCannotAddUser
 * @expectedExceptionMessage Cannot add user George
 */
function testWillNotAddAnAlreadyExistingUser() {
	$command = \Mockery::mock('SystemCommand');
	$command->shouldReceive('execute')->once()->with('adduser George')->andReturn(false);
	$command->shouldReceive('getFailureMessage')->once()->andReturn('User already exists on the system.');
	$userControl = new UserControl($command);
	$userControl->add('George');
}

We modified our test to expect our own custom exception, ExceptionCannotAddUser. The rest of the test is unchanged.

class ExceptionCannotAddUser extends Exception {

	public function __construct($userName, $reason) {
		$message = sprintf(
			'Cannot add user %s. Reason: %s',
			$userName, $reason
		);
		parent::__construct($message, 13, null);
	}
}

The class that implements our custom exception is like any other class, but it has to extend Exception. Using custom exceptions also provides us a great place to do all the presentation related string manipulation. Moving the concatenation here, we also eliminated presentation from the business logic and respected the single responsibility principle.

public function add($username) {
	if (!$username) {
		throw new Exception('User can not be empty!');
	}

	if(!$this->systemCommand->execute(sprintf('adduser %s', $username))) {
		throw new ExceptionCannotAddUser($username, $this->systemCommand->getFailureMessage());
	}
}

Throwing our own exception is just a matter of changing the old “throw” command to the new one and sending in two parameters instead of composing the message here. Of course all tests are passing.

PHPUnit 3.7.28 by Sebastian Bergmann.

..

Time: 18 ms, Memory: 3.00Mb

OK (2 tests, 4 assertions)

Done.

Catching Exceptions in Your MVC

Exceptions must be caught at some point, unless you want your user to see them as they are. If you are using an MVC framework you will probably want to catch exceptions in the controller or model. After the exception is caught, it is transformed in a message to the user and rendered inside your view. A common way to achieve this is to create a “tryAction($action)” method in your application’s base controller or model and always call it with the current action. In that method you can do the catching logic and nice message generation to suit your framework.

If you do not use a web framework, or a web interface for that matter, your presentation layer should take care of catching and transforming these exceptions.

If you develop a library, catching your exceptions will be the responsibility of your clients.


Final Thoughts

That’s it. We traversed all the layers of our application. We validated in JavaScript, HTML and in our models. We’ve thrown and caught exceptions from our business logic and even created our own custom exceptions. This approach to validation and exception handling can be applied from small to big projects without any severe problems. However if your validation logic is getting very complex, and different parts of your project uses overlapping parts of logic, you may consider extracting all validations that can be done at a specific level to a validation service or validation provider. These levels may include, but not need to be limited to JavaScript validator, backend PHP validator, third party communication validator and so on.

Thank you for reading. Have a nice day.

January 06 2014

18:43

Writing Robust Web Applications – The Lost Art of Exception Handling

As developers, we want the applications we build to be resilient when it comes to failure, but how do you achieve this goal? If you believe the hype, micro-services and a clever communication protocol are the answer to all your problems, or maybe automatic DNS failover. While that kind of stuff has its place and makes for an interesting conference presentation, the somewhat less glamorous truth is that making a robust application begins with your code. But, even well designed and well tested applications are often lacking a vital component of resilient code – exception handling.

I never fail to be amazed by just how under-used exception handling tends to be even within mature codebases. Let’s look at an example.


What Can Possibly Go Wrong?

Say we have a Rails app, and one of the things we can do using this app is fetch a list of the latest tweets for a user, given their handle. Our TweetsController might look like this:

class TweetsController < ApplicationController
  def show
    person = Person.find_or_create_by(handle: params[:handle])
    if person.persisted?
      @tweets = person.fetch_tweets
    else
      flash[:error] = "Unable to create person with handle: #{person.handle}"
    end
  end
end

And the Person model that we used might be similar to the following:

class Person < ActiveRecord::Base
  def fetch_tweets
    client = Twitter::REST::Client.new do |config|
      config.consumer_key        = configatron.twitter.consumer_key
      config.consumer_secret     = configatron.twitter.consumer_secret
      config.access_token        = configatron.twitter.access_token
      config.access_token_secret = configatron.twitter.access_token_secret
    end
    client.user_timeline(handle).map{|tweet| tweet.text}
  end
end

This code seems perfectly reasonable, there are dozens of apps that have code just like this sitting in production, but let’s look a little more closely.

  • find_or_create_by is a Rails method, it’s not a ‘bang’ method, so it shouldn’t throw exceptions, but if we look at the documentation we can see that due to the way this method works, it can raise an ActiveRecord::RecordNotUnique error. This won’t happen often, but if our application has a decent amount of traffic it’s occurring more likely than you might expect (I’ve seen it happen many times).
  • While we’re on the subject, any library you use can throw unexpected errors due to bugs within the library itself and Rails is no exception. Depending on our level of paranoia we might expect our find_or_create_by to throw any kind of unexpected error at any time (a healthy level of paranoia is a good thing when it comes to building robust software). If we have no global way of handling unexpected errors (we’ll discuss this below), we might want to handle these individually.
  • Then there is person.fetch_tweets which instantiates a Twitter client and tries to fetch some tweets. This will be a network call and is prone to all sorts of failure. We may want to read the documentation to figure out what the possible errors we might expect are, but we know that errors are not only possible here, but quite likely (for example, the Twitter API might be down, a person with that handle might not exist etc.). Not putting some exception handling logic around network calls is asking for trouble.

Our tiny amount of code has some serious issues, let’s try and make it better.


The Right Amount of Exception Handling

We’ll wrap our find_or_create_by and push it down into the Person model:

class Person < ActiveRecord::Base
  class << self
    def find_or_create_by_handle(handle)
      begin
        Person.find_or_create_by(handle: handle)
      rescue ActiveRecord::RecordNotUnique
        Rails.logger.warn { "Encountered a non-fatal RecordNotUnique error for: #{handle}" }
        retry
      rescue => e
        Rails.logger.error { "Encountered an error when trying to find or create Person for: #{handle}, #{e.message} #{e.backtrace.join("\n")}" }
        nil
      end
    end
  end
end

We’ve handled the ActiveRecord::RecordNotUnique according to the documentation and now we know for a fact that we’ll either get a Person object or nil if something goes wrong. This code is now solid, but what about fetching our tweets:

class Person < ActiveRecord::Base
  def fetch_tweets
    client.user_timeline(handle).map{|tweet| tweet.text}
  rescue => e
    Rails.logger.error { "Error while fetching tweets for: #{handle}, #{e.message} #{e.backtrace.join("\n")}" }
    nil
  end

  private

  def client
    @client ||= Twitter::REST::Client.new do |config|
      config.consumer_key        = configatron.twitter.consumer_key
      config.consumer_secret     = configatron.twitter.consumer_secret
      config.access_token        = configatron.twitter.access_token
      config.access_token_secret = configatron.twitter.access_token_secret
    end
  end
end

We push instantiating the Twitter client down into its own private method and since we didn’t know what could go wrong when we fetch the tweets, we rescue everything.

You may have heard somewhere that you should always catch specific errors. This is a laudable goal, but people often misinterpret it as, “if I can’t catch something specific, I won’t catch anything”. In reality, if you can’t catch something specific you should catch everything! This way at least you have an opportunity to do something even if it’s only to log and re-raise the error.

An Aside on OO Design

In order to make our code more robust, we were forced to refactor and now our code is arguably better than it was before. You can use your desire for more resilient code to inform your design decisions.

An Aside on Testing

Every time you add some exception handling logic to a method, it’s also an extra path through that method and it needs to be tested. It’s vital you test the exceptional path, perhaps more so than testing the happy path. If something goes wrong on the happy path you now have the extra insurance of the rescue block to prevent your app from falling over. However, any logic inside the rescue block itself has no such insurance. Test your exceptional path well, so that silly things like mistyping a variable name inside the rescue block don’t cause your application to blow up (this has happened to me so many times – seriously, just test your rescue blocks).


What to Do With the Errors We Catch

I’ve seen this kind of code countless times through the years:

begin
  widgetron.create
rescue
  # don't need to do anything
end

We rescue an exception and don’t do anything with it. This is almost always a bad idea. When you’re debugging a production issue six months from now, trying to figure our why your ‘widgetron’ isn’t showing up in the database, you won’t remember that innocent comment and hours of frustration will follow.

Don’t swallow exceptions! At the very least you should log any exception that you catch, for example:

begin
  foo.bar
rescue => e
  Rails.logger.error { "#{e.message} #{e.backtrace.join("\n")}" }
end

This way we can trawl the logs and we’ll have the cause and stack trace of the error to look at.

Better yet, you may use an error monitoring service, such as Rollbar which is pretty nice. There are many advantages to this:

  • Your error messages aren’t interspersed with other log messages
  • You will get stats on how often the same error has happened (so you can figure out if it’s a serious issue or not)
  • You can send extra information along with the error to help you diagnose the problem
  • You can get notifications (via email, pagerduty etc.) when errors occur in your app
  • You can track deploys to see when particular errors were introduced or fixed
  • etc.
begin
  foo.bar
rescue => e
  Rails.logger.error { "#{e.message} #{e.backtrace.join("\n")}" }
  Rollbar.report_exception(e)
end

You can, of course, both log and use a monitoring service as above.

If your rescue block is the last thing in a method, I recommend having an explicit return:

def my_method
  begin
    foo.bar
  rescue => e
    Rails.logger.error { "#{e.message} #{e.backtrace.join("\n")}" }
    Rollbar.report_exception(e)
    nil
  end
end

You may not always want to return nil, sometimes you might be better off with a null object or whatever else makes sense in the context of your application. Consistently using explicit return values will save everyone a lot of confusion.

You can also re-raise the same error or raise a different one inside your rescue block. One pattern that I often find useful is to wrap the existing exception in a new one and raise that one so as not to lose the original stack trace (I even wrote a gem for this since Ruby doesn’t provide this functionality out of the box). Later on in the article when we talk about external services, I will show you why this can be useful.


Handling Errors Globally

Rails lets you specify how to handle requests for resources of a certain format (HTML, XML, JSON) by using respond_to and respond_with. I rarely see apps that correctly use this functionality, after all if you don’t use a respond_to block everything works fine and Rails renders your template correctly. We hit our tweets controller via /tweets/yukihiro_matz and get an HTML page full of Matzs’ latest tweets. What people often forget is that it’s very easy to try and request a different format of the same resource e.g. /tweets/yukihiro_matz.json. At this point Rails will valiantly try to return a JSON representation of Matzs’ tweets, but it won’t go well since the view for it doesn’t exist. An ActionView::MissingTemplate error will get raised and our app blows up in a spectacular fashion. And JSON is a legitimate format, in a high traffic application you’re just as likely to get a request for /tweets/yukihiro_matz.foobar. Tuts+ gets these kinds of requests all the time (likely from bots trying to be clever).

The lesson is this, if you’re not planning to return a legitimate response for a particular format, restrict your controllers from trying to fulfill requests for those formats. In the case of our TweetsController:

class TweetsController < ApplicationController
  respond_to :html

  def show
    ...
    respond_to do |format|
      format.html
    end
  end
end

Now when we get requests for spurious formats we’ll get a more relevant ActionController::UnknownFormat error. Our controllers feel somewhat tighter which is a great thing when it comes to making them more robust.

Handling Errors the Rails Way

The problem we have now, is that despite our semantically pleasing error, our application is still blowing up in our users’ face. This is where global exception handling comes in. Sometimes our application will produce errors that we want to respond to consistently, no matter where they come from (like our ActionController::UnknownFormat). There are also errors that can get raised by the framework before any of our code comes into play. A perfect example of this is ActionController::RoutingError. When someone requests a URL that doesn’t exist, like /tweets2/yukihiro_matz, there is nowhere for us to hook in to rescue this error, using traditional exception handling. This is where Rails’ exceptions_app comes in.

You can configure a Rack app in application.rb to be called when an error that we haven’t handled is produced (like our ActionController::RoutingError or ActionController::UnknownFormat). The way you will normally see this used is to configure your routes app as the exceptions_app, then define the various routes for the errors you want to handle and route them to a special errors controller that you create. So our application.rb would look like this:

...
config.exceptions_app = self.routes
...

Our routes.rb will then contain the following:

...
match '/404' => 'errors#not_found', via: :all
match '/406' => 'errors#not_acceptable', via: :all
match '/500' => 'errors#internal_server_error', via: :all
...

In this case our ActionController::RoutingError would be picked up by the 404 route and the ActionController::UnknownFormat will be picked up by the 406 route. There are many possible errors that can crop up. But as long as you handle the common ones (404, 500, 422 etc.) to start with, you can add others if and when they happen.

Within our errors controller we can now render the relevant templates for each kind of error along with our layout (if it’s not a 500) to maintain the branding. We can also log the errors and send them to our monitoring service, although most monitoring services will hook in to this process automatically so you don’t have to send the errors yourself. Now when our application blows up it does so gently, with the right status code depending on the error and a page where we can give the user some idea regarding what happened and what they can do (contact support) – an infinitely better experience. More importantly, our app will seem (and will actually be) much more solid.

Multiple Errors of the Same Type in a Controller

In any Rails controller we can define specific errors to be handled globally within that controller (no matter which action they get produced in) – we do this via rescue_from. The question is when to use rescue_from? I usually find that a good pattern is to use it for errors that can occur in multiple actions (for example, the same error in more than one action). If an error will only be produced by one action, handle it via the traditional begin...rescue...end mechanism, but if we’re likely to get the same error in multiple places and we want to handle it the same way – it’s a good candidate for a rescue_from. Let’s say our TweetsController also has a create action:

class TweetsController < ApplicationController
  respond_to :html

  def show
    ...
    respond_to do |format|
      format.html
    end
  end

  def create
    ...
  end
end

Let’s also say that both of these actions can encounter a TwitterError and if they do we want to tell the user that something is wrong with Twitter. This is where rescue_from can be really handy:

class TweetsController < ApplicationController
  respond_to :html

  rescue_from TwitterError, with: twitter_error

  private

  def twitter_error
    render :twitter_error
  end
end

Now we don’t need to worry about handling this in our actions and they will look much cleaner and we can/should – of course – log our error and/or notify our error monitoring service within the twitter_error method. If you use rescue_from correctly it can not only help you make your application more robust, but can also make your controller code cleaner. This will make it easier to maintain and test your code making your application that little bit more resilient yet again.


Using External Services in Your Application

It’s difficult to write a significant application these days without using a number of external services/APIs. In the case of our TweetsController, Twitter came into play via a Ruby gem that wraps the Twitter API. Ideally we would make all our external API calls asynchronously, but we’re not covering asynchronous processing in this article and there are plenty of applications out there that make at least some API/network calls in-process.

Making network calls is an extremely error prone task and good exception handling is a must. You can get authentication errors, configuration problems, and connectivity errors. The library you use can produce any number of code errors and then there is a matter of slow connections. I am glossing over this point, but it’s oh so crucial since you can’t deal with slow connections via exception handling. You need to appropriately configure timeouts in your network library, or if you’re using an API wrapper make sure it provides hooks to configure timeouts. There is no worse experience for a user than having to sit there waiting without your application giving any indication of what’s happening. Just about everyone forgets to configure timeouts appropriately (I know I have), so take heed.

If you’re using an external service in multiple places within your application (multiple models for example), you expose large parts of your application to the full landscape of errors that can be produced. This is not a good situation. What we want to do is limit our exposure and one way we can do this is putting all access to our external services behind a facade, rescuing all errors there and re-raising one semantically appropriate error (raise that TwitterError that we talked about if any errors occur when we try to hit the Twitter API). We can then easily use techniques like rescue_from to deal with these errors and we don’t expose large parts of our application to an unknown number of errors from external sources.

An even better idea might be to make your facade an error free API. Return all successful responses as is and return nils or null objects when you rescue any sort of error (we do still need to log/notify ourselves of the errors via some of the methods we discussed above). This way we don’t need to mix different types of control flow (exception control flow vs if…else) which may gain us significantly cleaner code. For example, let’s wrap our Twitter API access in a TwitterClient object:

class TwitterClient
  attr_reader :client

  def initialize
    @client = Twitter::REST::Client.new do |config|
      config.consumer_key        = configatron.twitter.consumer_key
      config.consumer_secret     = configatron.twitter.consumer_secret
      config.access_token        = configatron.twitter.access_token
      config.access_token_secret = configatron.twitter.access_token_secret
    end
  end

  def latest_tweets(handle)
    client.user_timeline(handle).map{|tweet| tweet.text}
  rescue => e
    Rails.logger.error { "#{e.message} #{e.backtrace.join("\n")}" }
    nil
  end
end

We can now do this: TwitterClient.new.latest_tweets('yukihiro_matz'), anywhere in our code and we know that it will never produce an error, or rather it will never propagate the error beyond TwitterClient. We’ve isolated an external system to make sure that glitches in that system won’t bring down our main application.


But What if I Have Excellent Test Coverage?

If you do have well-tested code, I commend you on your diligence, it will take you a long way towards having a more robust application. But a good test suite can often provide a false sense of security. Good tests can help you refactor with confidence and protect you against regression. But, you can only write tests for things you expect to happen. Bugs are, by their very nature, unexpected. To use our tweets example, until we choose to write a test for our fetch_tweets method where client.user_timeline(handle) raises an error thereby forcing us to wrap a rescue block around the code, all our tests will have been green and our code would have remained failure-prone.

Writing tests, doesn’t absolve us of the responsibility of casting a critical eye over our code to figure out how this code can potentially break. On the other hand, doing this kind of evaluation can definitely help us write better, more complete test suites.


Conclusion

Resilient systems don’t spring forth fully formed from a weekend hack session. Making an application robust, is an ongoing process. You discover bugs, fix them, and write tests to make sure they don’t come back. When your application goes down due to an external system failure, you isolate that system to make sure the failure can’t snowball again. Exception handling is your best friend when it comes to doing this. Even the most failure-prone application can be turned into a robust one if you apply good exception handling practices consistently, over time.

Of course, exception handling is not the only tool in your arsenal when it comes to making applications more resilient. In subsequent articles we will talk about asynchronous processing, how and when to apply it and what it can do in terms of making your application fault tolerant. We will also look at some deployment and infrastructure tips that can have a significant impact without breaking the bank in terms of both money and time – stay tuned.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
(PRO)
No Soup for you

Don't be the product, buy the product!

close
YES, I want to SOUP ●UP for ...