Production

Logging

  • morgan for dev

express-logger

for prod

  • supports log rotation (every 24 hours the log is copied and a new one starts)
  1. switch(app.get('env')){
  2. case 'development':
  3. // compact, colorful dev logging
  4. app.use(require('morgan')('dev'));
  5. break;
  6. case 'production':
  7. // module 'express-logger' supports daily log rotation
  8. app.use(require('express-logger')({
  9. path: __dirname + '/log/requests.log'
  10. }));
  11. break;
  12. }

Handling Uncaught Exceptions

  • When Express executes route handlers, it wraps them in a try/catch block, so it isn’t an uncaught exception.
  • Express will log the exception on the server side, and the client will get an ugly stack dump
  • The server will still be stable, and other requests will continue to be served correctly.
  • Add error handler after all your routes
  1. // This is the only exceptions that Express can catch
  2. app.get('/fail', (req, res) => {
  3. throw new Error('Nope:');
  4. });
  5. app.use((err, req, res, next) => {
  6. console.error(err.stack);
  7. app.status(500).render('500');
  8. });
  9. // This won't be catched by Express
  10. app.get('/epic-fail', (req, res) => {
  11. process.nextTick(() => {
  12. throw new Error('Kaboom!');
  13. });
  14. });
  • process.nextTick(function) & setTimeout(function, 0) execute when Node is idle, asynchronously. This causes Node to lose context about the http request it was served from so it shuts down the server, because it’s in an undefined state.
  • Node can’t know the purpose of the function, or its caller. So it can no longer assume that any further functions will work correctly.
  • When an uncaught exception occurs the best thing is to stop the server and have a failover mechanism.
  • An easy failover mechanism can be a cluster.

Two mechanisms to restart servers after uncaught exceptions

Domains and uncaughtExceptions events

Domains

  • The most recent and recommended approach (2014)
  • Domains wrap the uncaught exceptions in a known context. This allows to gracefully shutdown the server
  1. app.use((req, res, next) => {
  2. // create a domain for this request
  3. var domain = require('domain').create();
  4. // handle errors on this domain
  5. domain.on('error', (err) => {
  6. console.error('DOMAIN ERROR CAUGHT\n', err.stack);
  7. try {
  8. // failsafe shutdown in 5 seconds
  9. setTimeout(() => {
  10. console.error('Failsafe shutdown.');
  11. process.exit(1);
  12. }, 5000);
  13. // disconnect from the cluster
  14. var worker = require('cluster').worker;
  15. if (worker) worker.disconnect();
  16. // stop taking new requests
  17. server.close();
  18. try {
  19. // attempt to use Express error route
  20. next(err);
  21. } catch (err) {
  22. // if Express error route failed, try
  23. // plain Node response
  24. console.error('Express error mechanism failed.\n', err.stack);
  25. res.statusCode = 500;
  26. res.setHeader('content-type', 'text/plain');
  27. res.end('Server error');
  28. }
  29. } catch(err) {
  30. console.error('Unable to send 500 response.\n', err.stack);
  31. }
  32. });
  33. // add the request and reponse objectes to the domain
  34. domain.add(req);
  35. domain.add(res);
  36. // execute the rest of the request chain in the domain
  37. domain.run(next);
  38. });
  39. // other middleware and routes go here
  40. var server = http.createServer(app).listen(app.get('port'), () => {
  41. console.log('Listening on port %d.', app.get('port'));
  42. });
  • domains are set pu as a middleware
Articles

uncaughtExceptions events

  • is going to be deprecated

Monitoring your app

  • use third-party monitoring services additionally to your server monitoring
  • if your server goes down it might not have time to send an SOS. 3rd party monitoring would help diversify the monitoring

Services (Uptime monitors)

UptimeRobot

  • alerts can go to email, SMS, Twitter, iPhone app
  • can monitor the return code from a single page (HTTP codes)

Pingdom

Site 24x7

Application Failures

  • Monitor keywords suc as “server failure”
  • handle gracefully with a “We’re sorry” message and ping the staff for help
  • A sophisticated approach could be AWS Simple Notification Service (SNS)

GetSentry

Airbrake