Microservice architectures bring significant benefits to software teams. Microservices allow a team to scale individual parts of their application independently. They provide huge benefits in resilience, too. An application with a memory leak in a troublesome spot is a big problem for a monolithic architecture. One bug takes down the whole system and leads to extended outages. With a microservice architecture, a memory leak only takes down one part of the swarm. But there are downsides to the microservice approach. For starters, microservices complicate deployment plans. While you can solve some of those complications with orchestration tools, it’s another layer of technology in your stack.
Microservice architectures are a proper pain to debug. Coding a logic error in one microservice often leads to failures in a different service. Determining the origin of a bug is an extended process of tracking down and identifying a root cause. The benefit of each slice of the application having its own server(s) is you can scale and silo those servers. The downside is when they break, figuring out exactly where takes much longer. In this post, we’ll talk about how to solve some of those problems with microservice logging and unlock the benefits while minimizing the cost.
Centralized Microservice Logging
The first and most important rule of microservice logging is those logs should go to a single place. It’s easy to rig your microservices to log to AWS Cloudwatch or Azure Monitor. And with a lot of work, you’ll get to a point where you have something that almost keeps your team in the loop. But it’s important not to underestimate this work. By default, those tools will throw logs from each service into their own bin. Determining a user received an error because of a null pointer exception in some business logic that originated in your authentication service is a journey. Out of the box, you’ll be spending hours tracing requests between different services. When you’re working on a critical outage, this is time you can’t afford to spend.
It’s likely you already knew the value of centralized logging, though. If your team isn’t using a centralized microservice logging solution, this should be your first step. You need to get all your logs into the same place, so you’re not trying to play Frogger with different logging silos. But, once you have those logs in the same place, what do you do?
Microservice Logging Tip #1: Correlate Between Services
Once all your logs are in one place, you’re going to notice something when debugging: they’re noisy. If you’re working on a high-traffic site, your application generates millions of log entries per hour. That’s too much to look through! Tracing down the bug from earlier is still just as tough, even though all your logs are in one place. To solve this problem, I like to include a unique identifier generated by the client making a request from your server. The identifier is passed between each service needed to complete the request. Now, if you’re trying to troubleshoot a bug, your first step is to identify the unique identifier passed with the request. It’s even better if your error handling includes the unique ID during the error log.
This one tip can save you countless hours debugging your application when you’re using a centralized microservice logging approach. If you take one thing away from these tips, this should be it: get your logs into one place and make it easy to trace a single request between microservices.
Tip #2: Get Everything Into One Place
I know, this seems like I’m repeating myself. But this is worth focusing on. Often, teams log their business application logic in one place and think they’ve covered all their bases. Not so fast, my friend. Microservice logging covers more than application logic. Modern applications include services covering a whole range of logic. They also cover the services coordinating those services, and the services covering those services, too! It’s important to log not just your application logic, but also logic for your container framework. And you need to make sure you’re logging your container orchestration systems, too. Then, you’ve got the systems that glue all these things together. Imagine a bug where one service can’t talk to your database, even though it’s sending the request and the database is receiving incoming connections. Maybe it’s because of a faulty network connector. Maybe it’s because someone changed a firewall rule without understanding the consequences.
Getting all these logs from disparate systems into one place once again makes troubleshooting issues simpler and faster. During critical outages, speed saves your business money.
Tip #3: Monitor Your Logs
It’s essential to have visibility into your logs. At every level, you need to see what your software tells you. But this isn’t not enough for a scaling microservice architecture. Remember, you’re generating millions of log entries per hour. Often, debugging an issue after the fact isn’t enough. By the time you know you have a problem, you’ve already lost someone’s business. Instead, you need to know as soon as an issue presents itself. This means automated monitoring of your microservice logging. When all your logs flow into one system, you can glean intelligence about your application’s running state in real time. If a single microservice starts reporting slowdowns or issues, you can see it. When those issues start impacting other services, you know instantly.
A good microservice logging strategy will have automated alerts to notify the team when things are going off the rails. That’s your cue to hop in before an outage affects customers. Instead of merely using logs to analyze problems after the fact, good microservice logging uses logs to tell you before things get out of hand.
Tip #4: Don’t Sleep on Search
Searching through millions of log entries by hand is tedious, even when time isn’t of the essence. A high-quality microservice logging solution needs a quick and easy way to search those logs. The key here is quick. While lots of logging options allow for simple text searching, you’re going to want more insight than what a simple grep or AWK call can give you. If that’s the best you can muster, your microservice logging solution is leaving you hanging.
Instead, as you think about microservice logging, you want to analyze how quickly and easily you can search for detailed information. Logs you can’t find don’t bring any benefit.
Finding the Right Microservice Logging Solution
You can build all these tools yourself. It’s possible to spend dozens or hundreds of hours configuring CloudWatch to meet all the tips we provided in this post. You’ll have to spend time touching each part of your system, and some parts will be particularly challenging. But you can do it. Or, you can let someone else do ththeat work for you. After all, the goal here is to make your logs more efficient, not spend all day getting them set up. That’s where an option like SolarWinds® Papertrail™ can make life easier. It lives up to every tip in this list and does more to keep your business running smoothly, like tracking live tails of logs from multiple services in one place. Just as microservices unlock the ability to scale your application, Papertrail unlocks logging your application. Find out what you’ve been missing with a free trial of SolarWinds Papertrail today.
Frustration-free log management. (It’s a thing.)Aggregate, organize, and manage your logs with Papertrail
This post was written by Eric Boersma. Eric is a software developer and development manager who’s done everything from IT security in pharmaceuticals to writing intelligence software for the US government to building international development teams for non-profits. He loves to talk about the things he’s learned along the way, and he enjoys listening to and learning from others as well.