UMBC chess director and CSEE Professor Alan Sherman just shared the final results of the 2013 Pan-American Intercollegiate Chess Team Championship which was held over the past four days in Lubbock Texas. UMBC place second after Webster Univ. and ahead of Illinois and Teach Tech.

The top four qualifying schools from the PanAms go on to compete in the Final Four of college chess, held in the spring. The winner is considered national champion and takes home the President’s Cup.

During the championship, UMBC defeated the University of West Indies, the University of Toronto, the University of Texas, Brownsville, Webster University C, and the University of Texas, Dallas A. UMBC’s only loss was to Webster University A, in round three. UMBC Team B won the Division V prize.

The UMBC Chess Team A for the 2013 Pan-Am Champioship had the following members.

• Board 1: GM Niclas “The Dark Knight” Huschenbeth (USCF rating 2610)
• Board 2: GM Akshayraj “The Indian Knight” Kore (2519)
• Board 3: M Levan “The Georgian Gangster” Bregadze (2469)
• Board 4: IM and WGM Nazi “The Black Widow” Paikidze (2378)
• Alternate: WGM Sabina “Sunshine” Foisor (2315)

See the UMBC Chess web site more information on UMBC’s chess program or contact Dr. Sherman at 410-963-4779 or Sorry, you need javascript to view this email address.

MindMeld analyzes your conversations and tries to offer helpful info from the Web. It’s a cool idea, but it doesn’t work as well as it needs to.

A new app called MindMeld listens to you talk and tries to butt in with helpful information. There’s a clever idea at the core of that, but I found that it often misunderstood words and failed to get the gist of a conversation, which made it more distracting than beneficial.

When you first get introduced to New Relic, it’s easy to be overwhelmed by all the features. But like with most tools, as you slowly climb the learning curve and get familiar with the features provided out of the box, you begin to wonder how it all hangs together under the hood and if there is more you can do with what you have at hand.

Today I will look at how New Relic actually monitors transactions and how you can hook into the process. We will take a quick glance at the support provided by New Relic for monitoring background jobs and we’ll also dig into New Relic custom metrics, how you would use them and the benefits you get from doing so. By the end you will have a deeper understanding of how New Relic works and be able to utilize it more fully. In other words, we will have a deeper knowledge of our tools—something every developer should strive for.

Let’s begin by taking a quick look at how New Relic actually hooks-in to track the performance of your application.

How New Relic Tracks Transactions and How You Can Do the Same

It may seem a little like magic, you include a gem in your Gemfile:

gem 'newrelic_rpm'

And somehow New Relic monitors all your code. Of course it is just code, so let’s see how New Relic actually instruments your app so it can begin monitoring it when the agent gem is required. We will do this in the context of a Rails 4 app.

The first place to look is newrelic_rpm.rb, which has the following relevant code in it:

...
    if Rails::VERSION::MAJOR.to_i >= 3
      module NewRelic
        class Railtie < Rails::Railtie
    
          initializer "newrelic_rpm.start_plugin" do |app|
            NewRelic::Control.instance.init_plugin(:config => app.config)
          end
        end
      end
    else
    ...
    

So a Railtie gets created when the Rails version is above three, this gets a NewRelic::Control singleton instance (when it is initialized) and calls init_plugin. When the NewRelic::Control instance is created it figures out what framework is running (Rails 4 in our case) and loads some relevant code; we can see this in new_relic/control/class_methods#load_framework_class. The init_plugin method that gets executed lives in new_relic/control/instance_methods. The interesting code here is:

...
    if Agent.config[:agent_enabled] && !NewRelic::Agent.instance.started?
      start_agent
      install_instrumentation
      load_samplers unless Agent.config[:disable_samplers]
    ...
    

The install_instrumentation call is the important one. The implementation lives in new_relic/control/instrumentation. Skipping over the boilerplate bits, this works out which instrumentation files it needs to load and then requires them one by one. In our case, it will load files under new_relic/agent/instrumentation/rails4. One of the files here is action_controller.rb, when this gets required it eventually executes the following code via some metaprogramming magic:

executes do
      class ActionController::Base
        include NewRelic::Agent::Instrumentation::ControllerInstrumentation
        include NewRelic::Agent::Instrumentation::Rails4::ActionController
      end
      NewRelic::Agent::Instrumentation::ActionControllerSubscriber \
        .subscribe(/^process_action.action_controller$/)
    end
    

And here we get to the crux of it: ActionController::Base (from which all your controllers inherit) gets a couple of modules included in it, the most important of which is NewRelic::Agent::Instrumentation::ControllerInstrumentation. This is the start of how New Relic begins to monitor all your controller actions as ‘transactions’. Of course, it’s a somewhat simplified view and we’re glossing over a lot of detail, but it gives you an idea of how New Relic monitors your code. The question is, how can you use this information?

Monitoring Custom Frameworks & Background Jobs

You’re highly unlikely to find yourself in a situation where you’re using a web framework that New Relic does not already have instrumentation for (in the Ruby world), but let’s say you did. Knowing what we know now, we can easily manually instrument the controller actions of this custom framework. If we have a controller like this:

class CustomController
      def custom_action
        ...
      end
    end
    

We can instrument it like so:

class CustomController
      include NewRelic::Agent::Instrumentation::ControllerInstrumentation
    
      def custom_action
        ...
      end
      add_transaction_tracer :custom_action
    end
    

Now your controller method will be tracked like a transaction in the same way Rails actions get tracked. Of course, if you’ve rolled your own web framework which includes database handling code, you will need to do extra work to instrument much of that code, to allow New Relic to monitor more than just controller actions. But the general idea remains sound.

The above pattern becomes more useful when you want New Relic to track background jobs in your application. You’re a lot more likely to have rolled some custom background job handling code than you are to have written your own web framework. In fact, we did just this with Tuts+ initially, although we’re now migrating to Sidekiq. If you’re using one of the well-known background jobs systems like Sidekiq, Resque or Delayed Job, New Relic already has instrumentation built in, but if you did roll your own, the above pattern is all you need to monitor your tasks.

For example, our custom Tuts+ background jobs were regular ruby classes that responded to the execute method, so all we have to do is this:

class SomeBackgroundJob
      include NewRelic::Agent::Instrumentation::ControllerInstrumentation
    
      def execute
        ...
      end
      add_transaction_tracer :custom_action, category: :task
    end
    

The last bit, category: :task, is to ensure that New Relic does not track it as a web transaction, but instead treats it as a background task and makes it appear under the background tasks tab in the New Relic UI. If we create a base class for all our jobs, we can put the instrumentation there and child classes will inherit it, so we don’t have to worry about doing the above in every job class.

Customizing a Transaction Even More

Interestingly enough, even the web transactions that New Relic automatically monitors are not sacrosanct. You can, for example, add some custom parameters to be sent to New Relic for the transaction that is currently executing (if you’ve switched on capturing parameters).

You can do this at any point during the transaction. All you have to do is call ::NewRelic::Agent.add_custom_parameters(:key => 'value') at any time and the parameters you pass in will get added to the parameter data you see inside New Relic. For example, if we had a controller that looked like this:

class HelloController < ApplicationController
      def index
        ::NewRelic::Agent.add_custom_parameters(:hello => 'world')
      end
    end
    

Slow transactions would give us the following:

This isn’t all we can do. We can segment a transaction while it’s running by renaming it. Let’s say we want to treat a transaction as special when it is performed by a particular user. You may do something like this:

class HelloController < ApplicationController
      def index
        new_relic_name = NewRelic::Agent.get_transaction_name
        if current_user.name == 'Joe Customer'
          NewRelic::Agent.set_transaction_name("#{new_relic_name} - Joe Customer")
        end
      end
    end
    

Now this transaction will be treated as a separate transaction in the New Relic UI:

Even default New Relic instrumentation has some room for customization, but sometimes, just like Captain Kirk, you just need more power. This is where custom metrics come in.

New Relic Custom Metrics & How They Are Useful

Back in the day, you would have used custom metrics to monitor things like external service communication and use of various common tools like Redis. These days, New Relic has better ways of monitoring that stuff, so what do we need custom metrics for? I’ve found custom metrics to be useful in four situations:

• monitoring code that New Relic can’t see
• monitoring code you don’t control
• monitoring scripts
• monitoring totally custom events

Let’s take a quick look at each of them.

Monitoring Code New Relic Can’t See

New Relic is pretty good at breaking down the performance of your various application methods in a transaction trace, but sometimes you will see something like this in a trace:

It seems there is application code that New Relic couldn’t instrument for some reason. What we can do is help New Relic (and ourselves) out with some custom metrics. We need to figure out which method New Relic had trouble monitoring and hook in some custom metrics to track how long this method took to execute. This will then appear in all subsequent traces. Let’s say we have a class with a method we want to monitor via custom metrics:

class Order
      def amount
        ...
      end
    end
    

We can begin to track the amount method like so:

require 'new_relic/agent/method_tracer'
    
    class Order
      include ::NewRelic::Agent::MethodTracer
    
      def amount
        ...
      end
      add_method_tracer :amount, 'Custom/amount'
    end
    

The second parameter to add_method_tracer is the name that this custom metric will get in the New Relic UI. Metric names are slash separated strings and all custom metrics should start with ‘Custom/’. You could, for example, name your custom metric as ‘Custom/<class_name>/<method_name>’. At this point you will begin seeing the amount method in your transaction traces, in the New Relic UI. But, what if our amount method is highly complex and we want to monitor parts of it which we suspect are slow? My advice is that you should refactor your method—it’s too big, but if you can’t do that, you can instrument random code like so:

class Order
      extend ::NewRelic::Agent::MethodTracer
    
      def amount
        ...
        self.class.trace_execution_scoped(['Custom/amount/complex_code']) do
          ...complex code...
        end
        ...
      end
    end
    

Now the instrumented part of the method will be reported separately in your transaction traces. If you’ve previously instrumented the method itself, your new ‘inner’ metric will be grouped under the previous one.

This is by far the most common way you’ll end up using custom metrics in your code, but let’s look at the others anyway.

Monitoring Code You Don’t Control

Often you may be using a library which, you suspect, is slowing down your application. New Relic won’t instrument any random gem for you by default, so what can you do? You can fork the gem and add some instrumentation using the method we saw above, but an even simpler solution exists—use initializers. Let’s say you’re using the foobar library which has a class Foo with a method bar that you suspect has some slow code in it. All you need to do is create an initializer foobar_instrumentation.rb, and put the following in it:

require 'new_relic/agent/method_tracer'
    
    Foo.class_eval do
      include ::NewRelic::Agent::MethodTracer
    
      add_method_tracer :bar
    end
    

As you can see the code is very similar to what we had above, New Relic will work out a sensible name for your new custom metric based on the class and method name and you will begin to see it in your transaction traces. Use this to work out if that suspicious library is actually making your code perform badly, but don’t keep this instrumentation around permanently. It spams unnecessary initializers in your Rails app and pollutes your New Relic UI with custom metrics that you don’t really need to track permanently.

Monitoring Scripts

One part of web applications that is often neglected is scripts. To paraphrase a presentation I did recently—they’re still production code and should be treated as such. You don’t want production code to be performing badly, especially if you execute it on an ongoing basis via cron jobs (or a similar method that is not a background job in your system), so we can use New Relic to figure out if your scripts are slow.

You may instrument your script code using custom metrics as described above. It won’t show up in transaction traces as it won’t be part of a transaction. What you will be able to do though, is create a custom dashboard from the metrics you collect which should give you an idea if your script is performing badly.

The other thing you can do is to treat your script as a type of background job and instrument it accordingly (include NewRelic::Agent::Instrumentation::ControllerInstrumentation and so on). It will get lumped together with other background jobs in the UI, but you don’t have to worry about custom dashboards.

The only caveat with scripts is this: New Relic only sends data over the wire periodically. With a one-off script that executes quickly you will need to ensure that the collected data actually gets sent, so you may need to shut down the New Relic agent manually. A good rule of thumb is to manually start the agent at the beginning of each script and to shut it down at the end:

require 'newrelic_rpm'
    
    ::NewRelic::Agent.manual_start
    
    ... codez ...
    
    ::NewRelic::Agent.shutdown
    

This way you won’t ever need to wonder why your data is not appearing in the UI.

Monitoring Totally Custom Events

One of the interesting things about New Relic is that it lets you take advantage of its UI and data aggregation facilities for metrics that have nothing to do with performance (in theory). For example, you may want to have some visibility on how often users sign up to your application, the frequency of sales, or the total amount users pay when they make purchases. These are more business metrics than performance ones, but if it’s too much trouble to track these separately you can use New Relic to do so.

New Relic lets you record custom metrics directly via two API calls:

• record_metric
• increment_metric

You can use record_metric to track any metric that has an amount and increment_metric is pretty self explanatory. So we may, for example, do this:

...
    def purchase(amount)
    ...
      ::NewRelic::Agent.record_metric('Custom/purchase_amount', amount)
      ::NewRelic::Agent.increment_metric('Custom/purchase_count')
    ...
    end
    ...
    

The only way for you to see these metrics in the UI would be to create some custom dashboards. I have to mention that this would be a somewhat “creative” use of the New Relic API as it’s designed with performance data in mind, but it is certainly a handy thing to know when you need to throw a quick dashboard together and don’t want to set up a bunch of extra infrastructure.

The Dangers of Monitoring Too Much

Of course, all this power has a cost. If you collect too many custom metrics, it can begin to slow down your application. It can also slow down the New Relic UI and make it difficult to interpret the data as New Relic will collapse similar metrics into a summary one. New Relic recommends keeping the number of custom metrics you collect below 2000. I have found that custom metrics are best used periodically. Instrument the code you need, use the instrumentation to solve the problem you’re having and then remove the instrumentation. This way you get to solve your performance issues and the number of custom metrics you use is unlikely to grow too high.

Conclusion

We’ve dug into the internals of the newrelic_rpm gem and have learned how to tell New Relic about code that you consider a web transaction. We’ve looked at how to alter transactions on the fly, how to monitor background jobs and the various situations where it makes sense to use custom metrics. There is much you can do with New Relic beyond the functionality it provides out of the box and you’re now much more able to utilize it to its full potential. But, there is always more to learn, like how to build custom dashboards out of the metrics that you capture, or how to monitor infrastructure with plugins. We’ll be covering those topics and more in subsequent articles, so make sure to check back often. And as always if you have a question, want to share your own New Relic story or just want to say hi, don’t forget to leave a comment.