CloudWatch Logs is an essential tool to implement operational best practices in AWS. The service provides storage, analysis and processing capabilities for log data from multiple sources, including on-premises servers and EC2 instances.

While it's a useful and scalable service, CloudWatch Logs has some intrinsic limits. These limits are related to concurrency, throttling, log data ingestion, timeouts and field parsing. Admins should note these limits before using CloudWatch Logs, and, when possible, work around them.

To export log data into CloudWatch Logs, applications call the PutLogEvents API, which uploads an array of log events, as a batch, into a log stream. The size of the batch is based on the number and size of submitted log events. Each log event can be a maximum size of 256 KB, and the total batch size can be a maximum of 1 MB. Since these limits cannot be increased, closely monitor sizes when sending log events into a log stream.

The CloudWatch Logs agent is a common and recommended method for exporting logs. The agent skips log events that exceed 256 KB in size. The agent also automatically adjusts the length of each batch to comply with the 1 MB restriction.

For high-volume applications, or tasks that require exporting a large amount of data into CloudWatch Logs, the PutLogEvents API has a limit of 5 requests per second per log stream. Even though this limit -- which cannot be increased -- is enough for most tasks, there are situations where a log export job can reach it. Create multiple log streams and ingest data concurrently to increase the throughput. In doing this, you avoid the limit and achieve the desired throughput.

CloudWatch Logs Insights is a useful tool for troubleshooting, as well as extracting operational data. It enables developers to analyze and aggregate log data using a predefined query language. A limitation to CloudWatch Logs Insights, however, is that queries time out after 15 minutes.

To avoid this, narrow the timestamp range to reduce the data window being analyzed. Alternatively, split the data into multiple log streams, so that queries cover smaller amounts of data and execute below the timeout limit. Then, add up the results from multiple query executions to cover the desired data range.

CloudWatch Logs Insights supports querying multiple log groups simultaneously. Although this is a useful feature to analyze log data, developers can access only up to 20 log groups in a single query -- a quota that cannot be increased.

If an application needs to query more than 20 log groups, split queries and then aggregate data from multiple queries. CloudWatch Logs Insights supports a maximum of 10 concurrent queries.

The CloudWatch Logs quotas mentioned in this article are some of the most relevant to cloud admins as of the date of publication. Reference AWS documentation for the full and up-to-date list of service quotas to avoid potential blocking issues when building and operating scalable cloud applications.

Metric filters are a CloudWatch Logs feature that let developers parse text in log data and convert patterns into CloudWatch metrics. This is a convenient way to monitor application events and automate actions such as notifications, alarms or custom logic. There is a limit of 100 metric filters per log group, which cannot be increased. This limit should be enough to implement a wide range of filters as a result of log events. Otherwise, it's possible to split data into multiple log groups to configure a higher number of metric filters.

Subscription filters allow incoming log data to integrate with other AWS services, such as Lambda, Kinesis Firehose or Kinesis Data Streams. There is a fixed quota that limits each log group to a maximum of two subscription filters. One way to work around this is to use Kinesis Streams or Lambda functions as an interface to integrate with more AWS services, beyond the quota in CloudWatch. This would allow log data to be exported to any number of AWS services.

There are also throttling limits in CloudWatch Logs APIs, including the following:

Each API has its own limit in terms of requests per second, which also varies by region.

Part of: Manage and optimize Amazon CloudWatch Logs

To improve the reliability of their applications, AWS users might want to consider exporting log data to the cloud for analysis and pattern detection.

Logs can reveal important information about your systems, such as patterns and errors. Learn how to search logs with CloudWatch Logs, key concepts and other options.

There are several reasons to export data out of CloudWatch Logs and into S3. Use this step-by-step tutorial to automate the process.

CloudWatch Logs imposes certain limits and service quotas related to log data ingestion and other areas. Fortunately, in many cases, there are workarounds.

CloudWatch Logs Insights helps organizations gain insights from a deluge of log data on applications and services. Learn how to implement key features.

Even with government money soon headed their way, top-tier semiconductor companies will still have to deal with a range of ...

Blockchain has been a significant contributor to the global chip shortage. Explore the role this rising technology has played.

Congress approved the CHIPS Act and billions more for scientific research to help the U.S. better compete against China in ...

When security incidents arise -- and they will -- IT administrators can restore, reset or completely wipe VMs to return them to ...

Dan Lorenc reflects on a whirlwind year for his open source project, now officially incorporated into Kubernetes and GitHub's npm...

AWS and other IT vendors will start building connectors based on a new standard schema meant to streamline data sharing between ...

Many organizations struggle to manage their vast collection of AWS accounts, but Control Tower can help. The service automates ...

There are several important variables within the Amazon EKS pricing model. Dig into the numbers to ensure you deploy the service ...

AWS users face a choice when deploying Kubernetes: run it themselves on EC2 or let Amazon do the heavy lifting with EKS. See ...

Background features and processes can often take up precious OS resources. With VMware's OS Optimization Tool, you can run ...

Learn how container management products from VMware, Red Hat and Hewlett Packard Enterprise compare when it comes to their ...

Submit your entry for the Best of VMware Explore 2022 Awards for a chance to win.

All Rights Reserved, Copyright 2010 - 2022, TechTarget Privacy Policy Cookie Preferences Do Not Sell My Personal Info