Advanced New Relic: Custom Events, NRQL Engineering, and Production Dashboard Design

Standard APM instrumentation tells you how your application performs. Custom instrumentation tells you how your business performs. This guide covers the techniques that separate basic New Relic users from engineers who use it as a genuine operational intelligence platform.

Custom Events: Beyond Default Telemetry

New Relic's default agents collect transactions, errors, and system metrics. Custom events let you push any structured data you care about — business events, third-party status checks, operational milestones — into New Relic and query it with NRQL.

Sending Custom Events via the Event API

const https = require('https');

function sendCustomEvent(accountId, licenseKey, eventType, attributes) {
  const payload = JSON.stringify([{ eventType, ...attributes }]);

  const options = {
    hostname: 'insights-collector.newrelic.com',
    path: `/v1/accounts/${accountId}/events`,
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-Insert-Key': licenseKey,
    },
  };

  return new Promise((resolve, reject) => {
    const req = https.request(options, (res) => {
      let data = '';
      res.on('data', (chunk) => (data += chunk));
      res.on('end', () => resolve(JSON.parse(data)));
    });
    req.on('error', reject);
    req.write(payload);
    req.end();
  });
}

// Usage: track third-party platform status
await sendCustomEvent(accountId, licenseKey, 'PlatformStatus', {
  platform: 'Vercel',
  status: 'operational',
  responseTime: 312,
  checkedAt: Date.now(),
});

Once ingested, PlatformStatus is a queryable event type just like Transaction or SystemSample.

NRQL on Custom Events

-- Track degradation over time
SELECT latest(status), average(responseTime)
FROM PlatformStatus
FACET platform
SINCE 24 hours ago
TIMESERIES 30 minutes

-- Alert when a platform goes non-operational
SELECT count(*)
FROM PlatformStatus
WHERE status != 'operational'
FACET platform
SINCE 10 minutes ago

NRQL Engineering: Patterns Worth Knowing

Funnel Analysis

SELECT funnel(session,
  WHERE pageUrl LIKE '%/signup%' AS 'Signup Page',
  WHERE pageUrl LIKE '%/verify%' AS 'Email Verify',
  WHERE pageUrl LIKE '%/dashboard%' AS 'Dashboard'
)
FROM PageView
SINCE 7 days ago

Percentile Distributions

Average response time hides the tail. Use percentiles:

SELECT percentile(duration, 50, 90, 95, 99)
FROM Transaction
WHERE appName = 'api-gateway'
SINCE 1 hour ago
TIMESERIES 5 minutes

The gap between p95 and p99 tells you whether your slowest responses are systemic or outliers. A large gap means a subset of users consistently has a poor experience.

Comparing Deployments

SELECT average(duration)
FROM Transaction
WHERE appName = 'checkout-service'
SINCE '2024-11-01 09:00:00'
UNTIL '2024-11-01 11:00:00'
COMPARE WITH 1 day ago

String Coercion for Reserved Keywords

If you FACET by a field whose name collides with a NRQL reserved keyword (like user or type), wrap it:

SELECT count(*) FROM Transaction FACET string(user) SINCE 1 hour ago

Synthetics Scripted API Monitors

Scripted API monitors run Node.js in New Relic's infrastructure, making HTTP requests and asserting on responses on a defined schedule. They are the right tool for monitoring third-party APIs, internal health endpoints, and multi-step authentication flows.

const assert = require('assert');

const statusEndpoints = [
  { name: 'Vercel', url: 'https://www.vercel-status.com/api/v2/status.json' },
  { name: 'Netlify', url: 'https://www.netlifystatus.com/api/v2/status.json' },
  { name: 'Cloudflare', url: 'https://www.cloudflarestatus.com/api/v2/status.json' },
];

async function checkPlatform({ name, url }) {
  const response = await $http.get({ url, timeout: 10000 });

  assert.equal(
    response.statusCode,
    200,
    `${name} status endpoint returned ${response.statusCode}`
  );

  const body = JSON.parse(response.body);
  const indicator = body?.status?.indicator;

  assert.notEqual(
    indicator,
    'major',
    `${name} is reporting a major outage`
  );

  $util.insights.set(`${name}_status`, indicator);
  $util.insights.set(`${name}_responseTime`, response.headers['x-response-time']);
}

for (const endpoint of statusEndpoints) {
  await checkPlatform(endpoint);
}

Schedule this every 5 minutes. Combine with a custom event push (see above) and you have a live third-party status feed queryable in NRQL.

Alert Condition Design for Production

Static vs. Anomaly vs. Baseline Conditions

Static conditions fire when a metric crosses a fixed threshold. Simple and predictable. Suitable for error rates, availability checks, and hard SLA limits.

Anomaly conditions fire when a metric deviates from its learned baseline by a configurable number of standard deviations. Suitable for traffic patterns with daily/weekly cycles where a fixed threshold would produce noise.

Baseline (NRQL) conditions give you full query control. Use them for custom events and business metrics where no pre-built condition type applies.

Alert Condition Anti-Patterns

Single-minute windows: A one-minute window on a volatile metric produces alert storms. Use 5-10 minute windows with a "for at least X minutes" setting for anything that shouldn't fire on transient spikes.

Alerting on averages: Average response time can look healthy while p99 is degraded. Alert on percentiles for user-facing services.

No priority differentiation: Every alert being Critical means no alert is Critical. Use a three-level hierarchy — Warning (awareness), Critical (immediate action), and Fatal (all-hands) — and enforce it consistently.

Dashboard Architecture for Production Operations

A production dashboard should answer specific operational questions, not display every metric available.

Layer Your Dashboards

L1 — Service Health (executive/on-call view): Availability, error rate, p95 latency, active incidents. One page, seven widgets maximum. This is the page someone opens at 2am to understand if there is a problem and how bad it is.

L2 — Service Deep Dive: Transaction breakdown by endpoint, slow queries, external service call performance, deployment markers. Used during active investigation.

L3 — Business Intelligence: Custom event data — user activity, feature adoption, conversion funnels, third-party dependency health. Used for operational reviews and planning.

Dashboard Filters and Variables

Use dashboard-level template variables to make a single dashboard serve multiple environments or services:

-- Variable: {{env}} = production | staging | development
SELECT average(duration)
FROM Transaction
WHERE environment = '{{env}}'
SINCE 1 hour ago

This eliminates dashboard proliferation — one well-designed dashboard replaces ten near-identical ones.

The Operational Maturity Test

You are using New Relic effectively when:

  • Your on-call engineers open a dashboard before opening a terminal
  • You can answer "which deployment caused this regression" in under two minutes using deployment markers and COMPARE WITH queries
  • Your alert policies have documented runbook links attached to every condition
  • Custom events capture business-level signals that no agent collects by default
  • Your Synthetics monitors detect third-party failures before your users report them

New Relic's ceiling is high. Most teams use 20% of its capability. The remaining 80% is where operational maturity lives.

Arivanandhan Chitheshwaran