Logs, The Easy Way

Your team invested countless hours in development.

Your QA people can barely keep their eyes open - they have worked so hard. Your lead developer who’s responsible for the deployment is almost dehydrated from so much pressure and sweat.

But it’s all worth it. Your app is live. Now everybody goes to sleep, and your pampered app, is all alone, serving your data to the entire world.

You forgot one thing - to give it a phone to call home, and tell you something went wrong.

You should have logs

Debugging is like being the detective in a crime movie where you are also the murderer.

— Filipe Fortes (@fortes) November 10, 2013

Arguing logs are important is like arguing automatic tests are important. Everybody knows that. Just that very few actually do it.

In reality the barrier is mostly because it’s hard to take the first step. In Drupal for example, there’s a watchdog that can send your errors to the DB, but it’s unrealistic to expect anyone to dig up an error in real time on multiple live apps.

Drupal also provides a syslog module that can later be used to forward the logs, however as many sites are hosted on platform solutions such as Pantheon, syslog isn’t always available.

In order to lower the barrier, Gizra has developed a general logging module that can send your watchdog data via HTTP called Logs HTTP. You just need to set the URL, and the severity level you would like to capture, and you’re done. This module can work for example on your logstash installation:

Since we want to concentrate on the actual development of the app (the same reason that brought us to use services like Pantheon in the first place) we’ve decided to go with Loggly, a cloud based log service. The price is reasonable and it provides many of the features we want, in particular the ability to send real time email alerts.

Logs HTTP module

The Logs HTTP module takes over your exception handler, so whenever an exception is thrown the backtrace (debug_backtrace()) of the request that caused the error is injected into the watchdog. This is super critical as it gives you a very clear insight to what caused the actual error. What triggered the error - that’s up to you to figure out.

All the events that are sent via HTTP, are sent in a PHP shutdown function, which means they won’t block your request. The module isn’t tightly coupled to the watchdog events, and you can use its API to send custom events.

As as single loggly account serves all our sites, we are using the “Unique ID” option, in order to easily understand on which server the error occurred. The Unique ID can be filled out manually, but it could also be environment specific. For example on Pantheon, we use this technique, that takes advantage of the $_ENV variable:

<?php
if (!empty($_ENV)) {
  // Add the URL only on Pantheon sites.
  $conf['loggly_http_url'] = 'Your-Loggly-URL';

  // Inject the site name and environment. This will result for example with ``my_site-live``
  $conf['loggly_http_uuid'] = $_ENV['PANTHEON_SITE_NAME'] . '-' . $_ENV['PANTHEON_ENVIRONMENT'];
}

The Outcome

In Gizra all our projects are connected to Loggly, where a simple alert has been setup - if an event with severity of error (or worth) was triggered - send the devs an email, so we know in real time we should start investigating an error.
Now all our apps and sites have a phone to call home.

Since Logs HTTP is an API module, you can still log specific events that are not in the severity level you have chosen, so for example, if a the severity level is “error”, but you want to log all the watchdog events of a login event you can:

<?php
/**
 * Implements hook_watchdog().
 */
function example_user_watchdog(array $log_entry) {
  if ($log_entry['message'] != 'Session opened for %name.') {
    return;
  }

  logs_http_register_event($log_entry);
}

And then in Loggly, you can have an alert with the query json.message:"Session opened for" NOT admin - which means you will be notified whenever a non admin logs in.

Conclusion

Like code review, automatic tests, and a proper deployment protocol, logging is an important part of the health of your application. With the Logs HTTP module, having this best practice implemented is only a few clicks away.