Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

September 25 2013

14:00

Getting Started With NewRelic in 30 Minutes

I remember working on a Rails app a few years ago and someone floated the idea of using this new service that had appeared on the scene. It was called NewRelic and they were promising to give you more insight into the performance of your Rails app, than you ever could get before. We gave it a try and it was impressive, more importantly it was something the Ruby web development ecosystem truly needed.

Fast forward to now and you’d be hard-pressed to find a Ruby web application that doesn’t have NewRelic hooked in. NewRelic as a company has continued to provide tools to monitor your Ruby apps, but they’ve also branched out into a number of other languages such as Java, Python and even .Net. But of course as the number of features you provide grows so does the complexity and the amount of documentation out there. It becomes hard to figure out where to start especially if you’re not yet an expert.

Today I thought we could go back to the roots of NewRelic and look at how we can get started with the service to monitor a Rails application.


A Basic Rails App

In order to use NewRelic we need something to monitor, so let’s set up a basic ‘Hello World’ Rails app.

The app we create will live under ~/projects/tmp/newrelic, and will be called newrelic_rails1. I assume you already have Rails installed:

cd ~/projects/tmp/newrelic
rails new newrelic_rails1
cd newrelic_rails1

There isn’t much for us to do to create our ‘Hello World’ app. We need a new controller:

rails g controller hello

Now we just need a route, we will get the root route of the application to use our controller. We also need a view, with the words ‘Hello World’. Given all this, our config/routes.rb should look like this:

NewrelicRails1::Application.routes.draw do
  root 'hello#index'
end

Our controller (app/controller/hello_controller.rb), will be as follows:

class HelloController > ApplicationController
  def index
  end
end

And our view (app/views/hello/index.html.erb), will be similar to:

<h1>Hello World!</h1>

We can now start up our development server:

rails s

When we curl localhost:3000, we get:

<!DOCTYPE html>
<html>
...
<body>
<h1>Hello World!</h1>
</body>
</html>

Everything is working!


Hooking in NewRelic

With Ruby it’s very simple. We add a gem to our Gemfile, run a bundle install, drop a config file into the config folder and we have all we need. In fact, NewRelic is pretty good at guiding you through this. All you need to do is log in to your account and if you haven’t deployed a NewRelic agent before, it’s pretty obvious what to do:

newrelic_start

Firstly, we install the NewRelic agent gem by adding it to our Gemfile, as per the instructions:

newrelic_install_gem

Our Gemfile will now look like this:

source 'https://rubygems.org'

gem 'rails', '4.0.0'
gem 'sqlite3'
gem 'sass-rails', '~> 4.0.0'
gem 'uglifier', '>= 1.3.0'
gem 'coffee-rails', '~> 4.0.0'
gem 'jquery-rails'
gem 'turbolinks'
gem 'jbuilder', '~> 1.2'

group :doc do
  gem 'sdoc', require: false
end

gem 'newrelic_rpm'

Whenever we add anything to the Gemfile we need to run:

bundle install

We also need a newrelic.yml, which you can download from NewRelic:

newrelic_config_download

It will come pre-configured with your license key. We need to put this file under config/newrelic.yml.

At this point if we ran our application in staging or production mode, we would already get data in our NewRelic account. So let us do so:

RAILS_ENV=production rails s

This time when we curl localhost:3000, we get:

<!DOCTYPE html>
<html>
<head><script type="text/javascript">var NREUMQ=NREUMQ||[];NREUMQ.push(["mark","firstbyte",new Date().getTime()]);</script>
  <title>NewrelicRails1</title>
  <link data-turbolinks-track="true" href="/stylesheets/application.css" media="all" rel="stylesheet" />
  <script data-turbolinks-track="true" src="/javascripts/application.js"></script>
  <meta content="authenticity_token" name="csrf-param" />
<meta content="i5rBPaG52bzM5Kn0SJwIbq6Qz0dG0KsIlcd8tb9vMV8=" name="csrf-token" />
</head>
<body>

<h1>Hello World!</h1>

<script type="text/javascript">if (typeof NREUMQ !== "undefined") { if (!NREUMQ.f) { NREUMQ.f=function() {
NREUMQ.push(["load",new Date().getTime()]);
var e=document.createElement("script");
e.type="text/javascript";
e.src=(("http:"===document.location.protocol)?"http:":"https:") + "//" +
  "js-agent.newrelic.com/nr-100.js";
document.body.appendChild(e);
if(NREUMQ.a)NREUMQ.a();
};
NREUMQ.a=window.onload;window.onload=NREUMQ.f;
};
NREUMQ.push(["nrfj","beacon-3.newrelic.com","b9119aa82e","2507356","cglYTRENCF4ERBtZB10KWRYKDABXGQ==",0,21,new Date().getTime(),"","","","",""]);}</script>

<p></body>
</html>

There is a bunch of JavaScript that got inserted into our pages so that NewRelic can monitor browser time. This is one way we can tell that our NewRelic integration is working. But it is not the only way, NewRelic also creates a log file:

% cat log/newrelic_agent.log

Logfile created on 2013-09-22 16:23:13 +1000 by logger.rb/36483
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Starting the New Relic agent in "production" environment.
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : To prevent agent startup add a NEWRELIC_ENABLE=false environment variable or modify the "production" section of your newrelic.yml.
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Reading configuration from config/newrelic.yml
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Enabling the Request Sampler.
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Environment: production
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Dispatcher: webrick
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Application: My Application
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Installing ActiveRecord 4 instrumentation
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Installing Net instrumentation
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Installing deferred Rack instrumentation
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Installing Rails 4 Controller instrumentation
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Installing Rails 4 view instrumentation
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Installing Rails4 Error instrumentation
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Finished instrumentation
[09/22/13 16:23:13 +1000 skorks-envato (12424)] INFO : Doing deferred dependency-detection before Rack startup
[09/22/13 16:23:16 +1000 skorks-envato (12424)] INFO : Reporting to: https://rpm.newrelic.com/accounts/303380/applications/2507356

We can also check our NewRelic account to make sure a new application has appeared for monitoring:

newrelic_monitored_application

There are however a few things that are not so nice:

  • Our application is named ‘My Application’
  • We accepted all the default configuration values, which may not suit our app
  • We had to launch our server in production mode (which is only possible cause it’s a brand new app that doesn’t rely on any external infrastructure)

So let us look at our newrelic.yml file in a little bit more detail to see how we can monitor our app performance exactly the way we want it.


Diving in to NewRelic Configuration

First of all, the NewRelic configuration file is extremely well commented and I encourage you to read the comments for the various configuration parameters to understand what all of them do.

Secondly, NewRelic configuration is environment aware, and configuration for all environments is defined in the one newrelic.yml file, this is very similar to, how the Rails database.yml file works. We define a bunch of common configuration values and then override the relevant ones in the specific environment blocks e.g.:

common: &default_settings
  license_key: '<your licence key>'
  app_name: My Application
  monitor_mode: true

  ...

development:
  <<: *default_settings
  monitor_mode: false

test:
  <<: *default_settings
  monitor_mode: false

production:
  <<: *default_settings
  monitor_mode: true

staging:
  <<: *default_settings
  monitor_mode: true

We can instantly begin to see how we can fix some of the points that we raised above. If we don’t want to have to launch our app in production mode while we’re tweaking our configuration, all we have to do is enable monitoring in development mode (we will need to remember to switch this off when we’re happy with our configuration as we don’t want development data cluttering up our NewRelic account).

development:
  <<: *default_settings
  monitor_mode: true

We should also override our application name for every environment that we have, to make sure they’re monitored separately and the application name makes sense:

  common: &default_settings
    license_key: '<your licence key>'
    app_name: newrelic_rails1
    monitor_mode: true

    ...

  development:
    <<: *default_settings
    monitor_mode: true
    app_name: newrelic_rails1 (Development)

  test:
    <<: *default_settings
    monitor_mode: false
    app_name: newrelic_rails1 (Test)

  production:
    <<: *default_settings
    monitor_mode: true
    app_name: newrelic_rails1 (Production)

  staging:
    <<: *default_settings
    monitor_mode: true
    app_name: newrelic_rails1 (Staging)
  

With just those configuration tweaks, when we start our server in development mode and curl localhost:3000:

newrelic_monitor_development

We’re now monitoring our application in development mode and our app name is what we expect. If your application is saying that it’s not receiving any data, give it a minute, it takes a little while for the data to start coming through.

The next most interesting (and often the most confusing) configuration value is the Apdex T-value. Unlike most of the other configuration parameters, this value does not live in the newrelic.yml file, but is instead found in the settings for the application within NewRelic:

newrelic_apdex

If you want to tweak your Apdex T-value you have to do it here, but what is this parameter and what is the right value to put in it? Well, NewRelic explains it in the following way:

Your application’s Apdex T-value is set to 0.5 seconds. That means requests responding in less than 0.5 seconds are satisfying (s), responding between 0.5 seconds and 2.0 seconds are tolerating (t), and responding in more than 2.0 seconds are frustrating (f).

Essentially, NewRelic uses the Apdex value to gauge the health of your application as far as performance is concerned, so if many of the requests that are monitored by NewRelic take longer than your Apdex value, NewRelic will consider your application to be performing poorly and if you’ve set up alerts, will notify you of the fact. Basically, you have to figure out, how fast you want each server request to be fulfilled by your application, so if you’re OK with a backend request taking two seconds, you can set your Apdex value to 2.0, but if you need a response to be returned within 100ms then you should set your Apdex value to 0.1.

If you have a new application you may set the Apdex value to the performance you desire from your application. If your app is an existing one, you may have some metrics regarding how fast it is/should be performing, and you can be guided by that. All requests which are fulfilled by the server in less than the Apdex T-value will be considered by NewRelic to be fine. All requests fulfilled within Apdex * 4 seconds will be considered tolerating (i.e. users can tolerate it). All responses that take longer than Apdex * 4 will be considered frustrating (frustrated users don’t tend to stick around). So, set your Apdex T-value in such a way that you actually get useful information out of it, the actual value depends on your domain and what you want to achieve (in terms of performance), there is no right or wrong answer.

We will set our Apdex T-value to 100ms (0.1), since all we have is a ‘Hello World’ app, and it should be able to return a response very quickly (even in development mode).


Even More NewRelic Configuration

It was a little funny that most of the configuration comes from the newrelic.yml file, but the Apdex T-value is in the application settings, so NewRelic now allows you to move all the configuration values from the YAML file into NewRelic:

newrelic_move_configuration

The advantage of this is that you don’t have to redeploy every time you want to tweak your configuration values, so it is definitely something worth considering. We will stick with the YAML file for now.

So what are some of the other useful NewRelic parameters we should know about?

Well, there is a set of parameters dealing with the NewRelic agent log file:

log_level: info
log_file_path: 'log'
log_file_name: 'newrelic_agent.log'

These have sensible defaults, but if we want the log file to go to a specific place or if we want to see more or less info in the file, we can easily control this. Since we’re just setting up NewRelic we will set the log level to debug, to make sure we don’t miss any important information (when we deploy we may want to set it to warn, or even error).

We now get a wealth of information in the log file, which (if read carefully) can give us a lot of insights into how NewRelic works:

% cat log/newrelic_agent.log</p>

[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Starting the New Relic agent in "development" environment.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : To prevent agent startup add a NEWRELIC_ENABLE=false environment variable or modify the "development" section of your newrelic.yml.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Reading configuration from config/newrelic.yml
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Not in Rake environment so skipping blacklisted_rake_tasks check: uninitialized constant Rake
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Updating config (add) from NewRelic::Agent::Configuration::YamlSource. Results:
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : {...}
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Not in Rake environment so skipping blacklisted_rake_tasks check: uninitialized constant Rake
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Updating config (add) from NewRelic::Agent::Configuration::ManualSource. Results:
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : {...}
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Not in Rake environment so skipping blacklisted_rake_tasks check: uninitialized constant Rake
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Installed New Relic Browser Monitoring middleware
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Installed New Relic Agent Hooks middleware
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Agent is configured to use SSL
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Using JSON marshaller
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Transaction tracing threshold is 2.0 seconds.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Ignoring errors of type 'ActionController::RoutingError'
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Ignoring errors of type 'Sinatra::NotFound'
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Errors will be sent to the New Relic service.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Ignoring errors of type 'ActionController::RoutingError'
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Ignoring errors of type 'Sinatra::NotFound'
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : RequestSampler max_samples set to 1200
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Resetting RequestSampler
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Enabling the Request Sampler.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Environment: development
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Dispatcher: webrick
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Application: newrelic_rails1 (Development)
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : EnvironmentReport failed to retrieve value for "Plugin List": undefined method `plugins' for #<Rails::Application::Configuration:0x007fb232401a00>
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : EnvironmentReport failed to retrieve value for "JRuby version": uninitialized constant NewRelic::EnvironmentReport::JRUBY_VERSION
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : EnvironmentReport failed to retrieve value for "Java VM version": uninitialized constant NewRelic::EnvironmentReport::ENV_JAVA
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : EnvironmentReport ignoring value for "Rails threadsafe" which came back falsey: nil
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Creating Ruby Agent worker thread.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Creating New Relic thread: Worker Loop
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : New Relic Ruby Agent 3.6.7.152 Initialized: pid = 12925
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Connecting Process to New Relic: bin/rails
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Not in Rake environment so skipping blacklisted_rake_tasks check: uninitialized constant Rake
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Created net/http handle to collector.newrelic.com:443
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Sending request to collector.newrelic.com:443/agent_listener/12/1f69cbd2a641bde79bdb5eb4c86a0ab32360e1f8/get_redirect_host?marshal_format=json
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Installing ActiveRecord 4 instrumentation
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Installing Net instrumentation
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Installing deferred Rack instrumentation
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Installing Rails 4 Controller instrumentation
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Installing Rails 4 view instrumentation
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Installing Rails4 Error instrumentation
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Finished instrumentation
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Registered NewRelic::Agent::Samplers::CpuSampler for harvest time sampling.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : Registered NewRelic::Agent::Samplers::MemorySampler for harvest time sampling.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : NewRelic::Agent::Samplers::ObjectSampler not supported on this platform.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] DEBUG : NewRelic::Agent::Samplers::DelayedJobSampler not supported on this platform.
[09/22/13 17:23:39 +1000 skorks-envato (12925)] INFO : Doing deferred dependency-detection before Rack startup
[09/22/13 17:23:40 +1000 skorks-envato (12925)] DEBUG : Uncompressed content returned
[09/22/13 17:23:40 +1000 skorks-envato (12925)] DEBUG : Created net/http handle to collector-1.newrelic.com:443
[09/22/13 17:23:40 +1000 skorks-envato (12925)] DEBUG : Sending request to collector-1.newrelic.com:443/agent_listener/12/1f69cbd2a641bde79bdb5eb4c86a0ab32360e1f8/connect?marshal_format=json
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Uncompressed content returned
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Server provided config: {...}
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Not in Rake environment so skipping blacklisted_rake_tasks check: uninitialized constant Rake
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Updating config (add) from NewRelic::Agent::Configuration::ServerSource. Results:
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : {...}
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Wiring up Cross Application Tracing to events after finished configuring
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Connected to NewRelic Service at collector-1.newrelic.com
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Agent Run       = 575257565.
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Connection data = {...}
[09/22/13 17:23:42 +1000 skorks-envato (12925)] INFO : Reporting to: https://rpm.newrelic.com/accounts/303380/applications/2507376
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Browser timing header: "<script type=\\"text/javascript\\">var NREUMQ=NREUMQ||[];NREUMQ.push([\"mark\",\"firstbyte\",new Date().getTime()]);</script>"
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Browser timing static footer: "if (!NREUMQ.f) { NREUMQ.f=function() {\nNREUMQ.push([\"load\",new Date().getTime()]);\nvar e=document.createElement(\"script\");\ne.type=\"text/javascript\";\ne.src=((\"http:\"===document.location.protocol)?\"http:\":\"https:\") + \"//\" +\n  \"js-agent.newrelic.com/nr-100.js\";\ndocument.body.appendChild(e);\nif(NREUMQ.a)NREUMQ.a();\n};\nNREUMQ.a=window.onload;window.onload=NREUMQ.f;\n};\n"
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Real User Monitoring is using JSONP protocol
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Reporting performance data every 60 seconds.
[09/22/13 17:23:42 +1000 skorks-envato (12925)] DEBUG : Running worker loop
[09/22/13 17:23:50 +1000 skorks-envato (12925)] DEBUG : Attempting to insert RUM header at beginning of head.

For example we can see that:

  • We can switch off monitoring even if it’s switched on in the configuration file, by setting an environment variable NEWRELIC_ENABLE=false
  • We can see that NewRelic inserts a bunch of Rack middleware
  • We’re using Webrick as our server, which is obviously in development mode, but in production it would be good to confirm that NewRelic recognises the server that we’re using
  • NewRelic is sending data to collector.newrelic.com:443
  • NewRelic is sending data every 60 seconds
  • Real user monitoring is done via JSONP

Very useful information when you’re trying to figure out how things hang together.

Most of the other configuration parameters are pretty self explanatory e.g.:

browser_monitoring:
  auto_instrument: true
capture_params: false

The only other one to possibly be aware of is:

transaction_tracer:
  transaction_threshold: apdex_f

The transaction tracer captures detailed data about requests that take too long. The transaction threshold is normally a multiple (x4) of the Apdex value, but it is often useful to divorce these values from each other. You might be happy with an Apdex score of one second, but you may want to capture detailed data about requests that take 1.5 seconds or longer (instead of the four seconds or longer which would happen by default). So you can set this parameter separately:

transaction_tracer:
  transaction_threshold: 1.5

The NewRelic Developer Mode

One of the configuration values you may have noticed was:

developer_mode: true

This should only be switched on in development (if at all). In development mode, NewRelic agent will store performance data about the last 100 requests in memory. You can look at this data at any time by hitting the /newrelic endpoint of your running application:

newrelic_developer_mode

I hardly ever use it, but it’s there if you need it.


Notifying NewRelic of Deployments

Whenever you’re working on the performance of your application, it’s always good to know if a particular deploy has had a positive or negative effect on performance. For this purpose, you can notify NewRelic every time you perform a deploy. This way if performance degrades or improves, you’ll be able to see which deploy was the culprit. NewRelic provides Capistrano hooks to do this, but I prefer the command line way:

% newrelic deployments -a 'newrelic_rails1 (Development)' -e 'development' -u 'skorks' -r 'abc123'
Recorded deployment to 'newrelic_rails1 (Development)' (2013-09-22 18:19:13 +1000)

The key thing is to correctly supply the application name as configured in the newrelic.yml file.

We will get nice lines on the relevant NewRelic graphs to indicate when a deployment occurred.

newrelic_deployment

Conclusion

You now know a whole lot about how NewRelic works and how to start using it to monitor a Rails application. But configuring things properly is only half the battle, what kind of metrics will NewRelic actually capture for you? And how can you use them to improve the performance of your application? We will look at some of these in a subsequent article. For now, have a go at configuring NewRelic for your Rails application (you’ll get a free T-shirt) and if you have any questions don’t forget to leave a comment.

February 14 2012

12:00

Attention Developers: NewRelic is your Secret Weapon

While the title of this article may sound like a cliche, hatched in the bowels of PR hell, I’m serious when I say that NewRelic is your secret weapon.

In this article, I’ll talk about the common aspects of web application performance, and then demonstrate how NewRelic makes it blissfully easy to manage.

In the nearly six years that I’ve worked at Envato – previously developing and managing the series of marketplaces and currently managing development of the Tuts+ blog network – I’ve used NewRelic to keep costs down, improve and debug performance problems from the small to the large, and avert potential catastrophes.

If you’re new to this topic, or don’t currently manage a website for someone in any capacity, don’t stress; this article will still be useful. You never know when this knowledge could save your bacon, and I’d wager it’s inevitable that it will – unless, of course, you decide to throw in the hypothetical developer towel and become an astronaut or rancher.


NewRelic in 20 Seconds

Before I launch into a reasonably long tirade on web application performance, it makes sense to quickly sum up what NewRelic is before you trundle off to Reddit or something similar.

NewRelic is a managed service (SaaS) that you “plug in” to your web app, which collects and aggregates performance metrics of your live web application.

The information it provides can help you find answers to questions like: Is my website slow? Who is it slow for? Where is it slow exactly? Do we need more, or bigger servers? What can we do to improve things?

These questions and their answers are oftentimes crucial to your web application’s success or failure. If you’ve never collected performance metrics from your live web app, you’re literally running while blindfolded; at some point you’re going to hit a wall!

Before I take you on a tour of NewRelic’s features, I have to define what web application performance is. Let’s get to it.


What Exactly is Web Application Performance?

Front-end performance is about “perceived” performance.

I like to split web app performance into two conceptual parts: front-end performance and back-end performance. While the two areas do indeed have crossover and affect one another, it’s helpful to draw a distinction.

Primarily, you can think of front-end performance as areas concerning perceived performance, such as how long a page takes to fully render to an end user. Variables that affect this type of performance include:

  • How large your HTML, CSS, JavaScript and images are
  • How many HTTP requests are sent to servers to fetch all of these assets
  • How they are organized in the page to affect perceived performance, whether or not a user’s browser has to re-download assets regardless if they’re the same or not.

I have only seen web applications and websites “fall over” as a result of mismanagement of the back-end.

Back-end performance involves some kind of programming language that runs your code (i.e. PHP, Ruby), and some kind of database server (i.e. MySQL). Generally, most web applications are assembling HTML documents to be sent to your user’s browser, and are made up of data fetched from one or more databases – or even one or more external service (such as the Twitter API). I also typically lump in server resources (such as CPU usage, memory usage, disk IO) into this category, as it’s the code running on your server (not in your user’s browser) that affects these resources.

Why is this distinction so important? Because, in my experience, I have found that a confusion between the two leads to useless effort being applied when trying to improve performance issues. I have witnessed work on the front-end performance of ailing websites when the actual issue has been the backend. On the other hand, I have watched people focus on back-end optimization when the problem has been on the front-end. It’s essential that you understand and appreciate the difference.

On their own, these two subjects can be rather deep and complicated, and it’s a topic for an entirely different series of posts. While I’m decidedly specialized in back-end performance, in all of my professional career, I have only seen web applications and websites “fall over” as a result of mismanagement of the back-end.


Three and a Half Approaches to Managing Performance

There are three ways in which people tend to manage the performance of their web applications:

  1. Write code, deploy it, and hope for the best.
  2. Write code, guess which areas will become bottlenecks, measure and optimize them up front, deploy and hope for the best.
  3. Write code, measure the live application with something like NewRelic, then fix and tune as appropriate.

The first approach is 100% reactionary. If you follow this method, you will only know if your web app is failing or performing poorly when your customers tell you (if they ever tell you).

The second approach is considerably more mature; the developers are preempting problems and attempting to resolve them upfront. While this is admirable, the possibility of spending vital resources optimizing the wrong area, and the lack of ongoing feedback will provide few facts about what is truly going on in the live environment.

The third approach is the almost ideal situation. By monitoring a live web application, you’re able to review how various things are performing, based on what your customers are actually doing. You can write code and receive immediate feedback on how well (or not) it’s performing.

The Ideal Approach

The ideal approach is to follow the third and apply a healthy measure of the second.

It doesn’t hurt to consider performance up front; it is infinitely more useful to have true metrics. The old programming adage, “premature optimization is the root of all evil” may apply here, though, in programming, as in life, axioms are rarely anything more than half-truths.


Measurement & Management: A Balancing Act

There is no such thing as one true method to managing your web application’s performance.

No matter what anyone says (including me!), there is no such thing as one true method to managing your web application’s performance. Depending on your app and customers, there will be different approaches and techniques. Yet one thing remains constant: you have to measure.

So, what do you measure? Again, there isn’t one true list, yet there’ll always be a common number of metrics worth measuring. For example:

  • The number of application requests over time.
  • The wall clock time requests take to complete.
  • The CPU usage of your servers over time.
  • The hard drive reads, writes and utilization over time (known as Disk IO).
  • The number of database queries, and the time they take to run.
  • Queries run on your database that take over two seconds to complete (slow queries).
  • Incoming and outgoing bandwidth over time.

This list, while certainly not exhaustive, will offer significant insight into your web app’s behavior, especially if you’ve never monitored them previously.

Once you have this kind of data, the management of your web application is where all the fun begins. You may find that, once removing a bottleneck in your database (perhaps a few slowly executed queries), you’ll expose another as more server resources are freed up. It truly is a balancing act.

Ultimately, what successful management looks like is something like this: you may double the efficiency of that single server of yours, allowing you to delay purchasing a second. On a larger scale, you may cut your server farm down by a factor of 50%, and on a large enough scale, that can equate to serious money. On a lighter side, you may simply provide good quality of service to your customers with no sudden surprises.


NewRelic: Your Secret Weapon

Now that we’ve covered the “what” and “how” bits, let’s take a look at NewRelic. Once upon a time in software-land, you had to roll your own measurement into an app – if you measured at all (which can be as much work as writing your app itself). NewRelic allows you to simply plug in its agent to your Ruby, PHP, .NET or Python application, and begin collecting real data right away.

Thoughtfully, their product is split up into three core regions:

  • End user monitoring (front-end, the browser)
  • Application monitoring (back-end, your code)
  • Server monitoring (back-end, the servers)

Let’s have a look at each, in the order they were historically released.

The very first feature NewRelic launched was application monitoring. It tracks and reports on ‘Requests per Minute’ (aka RPM), average response times of these requests, and keeps this data for you to analyze. This is particularly useful for discovering trends in traffic over time (e.g. does my site get slower as our page views increase?).

Additionally, the “slow transaction traces” will provide you with a list of recent requests from real users that were disproportionately slow. Inspecting these allows you to drill down and determine why a request took such a long time, giving you the information you need to improve it.

End user monitoring will provide you with insight into how your site is rendering in the users’ browsers. It breaks the total time into chunks, based on things like network time (how long assets took to download), DOM rendering (how long your browser spend figuring out your HTML), image loading (as served by your web server or a CDN).

A neat feature of end user monitoring is that it shows you how well or poorly your application is performing for users in different countries. For example, perhaps 50% of your customers are based in the UK, while the other 50% are in the US. You might discover that front-end performance isn’t too great in the UK, due to the physical distance from your servers. Introducing a CDN or a server in the UK will improve their experience.

The best part of using NewRelic and taking action based on its data is that, once you’ve made any number of changes, you can immediately review if the changes have been effective or not!

The last piece of the puzzle, and the most recent monitoring NewRelic has introduced, is their server monitoring tools. I’ve always remarked that you must correlate your server’s resources with your web application response times to get a fuller picture of efficiency. You may have excellent response times, but you also may be needlessly sacrificing significant server resources to provide them.

I have seen apps with excellent YSlow scores, for instance, but absolutely no headroom for more traffic – even on significant amounts of hardware!

I hope by now you’re starting to see how valuable this kind of information is!


Installing NewRelic

You’ll need to at least be on a VPS and have root access for the PHP agent.

One of my only criticisms of NewRelic is that it’s not easy to install for some types of users. If you are a Ruby on Rails programmer, you’ll find it fairly easy, as it’s a simple Rails plugin.

If you’re a PHP developer and aren’t comfortable goofing around on the command line, you’re going to find it difficult to install, as it’s a PHP extension and requires a daemon to be installed running alongside your web server. However, some PHP cloud platforms, like PHPFog offer NewRelic integration out of the box.

This is unfortunate in my mind, as it’s a hurdle for most people. I hope NewRelic are currently looking to partner with more commodity web hosting providers, so that their product is more accessible to a wider audience. There’s literally no tool like it on the market at present, and they should be making it easy for all PHP developers to use.

If you’re using existing hosting, you’ll need to at least be on a VPS and have root access for the PHP agent. Being completely fair, to spin up a VPS from a provider like Linode, and installing Apache, PHP, MySQL and NewRelic is a short process, but it does requires some comfort and know-how in a shell.

The best way to get started using PHP and NewRelic is to use a tool like Oracle VirtualBox, install Linux, set up Apache and PHP and then install the agent. Then you’ll be able to use NewRelic in your local development environment, at the very least.

I personally haven’t had any experience with the Python agent, and I’ve heard third-hand that the .NET component is easy as pie to get up and running.


How Envato Uses NewRelic

Envato has been using NewRelic since 2008. We’ve used it in the following products with good (and sometimes interesting) results:

The Marketplaces

Initially, we discovered roughly three major slow spots in unexpected places in the marketplaces. We discovered what our highest trafficked requests were, and focused on optimizing them specifically. If 80% of our time was spent in one spot, making it twice as fast increased capacity and saved us from allocating more funds to hardware. We’ve spotted unusual traffic (such as spammers and hackers) allowing us to take precautionary measures sooner than later, thus improving the experience for our real customers. We use it daily to monitor the performance of all our new and existing code.

The Tuts+ Blogs

In 2009-2010, Envato’s blog network had serious stability problems due to a number of architectural problems. It was my job to step in and solve the issues. After performing an architectural analysis and a redesign of it, we plugged in the (then beta!) PHP monitor. We discovered many, many undesirable things!

  • 20% of requests were hits to feeds (which should have been cached or sent to FeedBurner)
  • 3 SQL queries were routinely taking more than 5 seconds to return results
  • Long-running WP-Cron tasks were tying up our web worker pool
  • 404 pages were taking more than 1 second to generate!

Over the course of 2010-2011, we progressively sorted out the issues until they were, more or less, all solved. To this day, we still monitor the PHP blogs using NewRelic. And now, thankfully, the blog network is nice and stable.

The Tuts+ Premium Redesign

When we launched the Tuts+ Premium redesign, we used NewRelic to debug performance problems before the actual launch, on the actual servers they were to run on. This allayed any fears of disaster upon launch. We continue to monitor the site’s performance, using NewRelic.

Today, any important application at Envato has a NewRelic agent plugged in. It honestly has saved us heaps of time and money, and allowed us to provide quality of service to our users and customers.


Other Tools Envato Uses to Augment NewRelic

It wouldn’t be fair to not mention the other tools we use to look after our applications. We currently use ScoutApp for finer-grained server monitoring (it supports user contributed ‘plugins’ so we can monitor specific services like HAProxy, Nginx, etc). We also use AirBrake which logs and aggregates our errors in our Ruby on Rails applications.

Lastly, we have rolled some of our own specialized, custom tools that check things like cache hits, backend requests, revenue, sign ups, notifications when a significant deviation from the trends occur. For example, revenue halting or dropping might mean our payment integration is broken; a change in sign ups means we might have been targeted by spammers creating ghost accounts for later use.


Wrapping Up

If you work on any kind of web application that is business critical, or are tasked with fixing a not-quite-working app, NewRelic is going to be invaluable to you.

If you have any questions, ask away in the comments and I’ll do my best to answer them. Particularly, if there’s interest in a screencast on how to set up a VPS or VM with NewRelic, I’m sure we could arrange one for you.

Become a programmer superhero; use NewRelic!


Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
Get rid of the ads (sfw)

Don't be the product, buy the product!

Schweinderl