|
As we speak we’re increasing Amazon CloudWatch capabilities to unify and handle log knowledge throughout operational, safety, and compliance use circumstances with versatile and highly effective analytics in a single place and with decreased knowledge duplication and prices.
This enhancement implies that CloudWatch can robotically normalize and course of knowledge to supply consistency throughout sources with built-in assist for Open Cybersecurity Schema Framework (OCSF) and Open Telemetry (OTel) codecs, so you possibly can deal with analytics and insights. CloudWatch additionally introduces Apache Iceberg appropriate entry to your knowledge by means of Amazon Easy Storage Service (Amazon S3) Tables, with the intention to run analytics, not solely regionally but in addition utilizing Amazon Athena, Amazon SageMaker Unified Studio, or every other Iceberg-compatible instrument.
You may as well correlate your operational knowledge in CloudWatch with different enterprise knowledge out of your most well-liked instruments to correlate with different knowledge. This unified method streamlines administration and gives complete correlation throughout safety, operational, and enterprise use circumstances.
Listed below are the detailed enhancements:
- Streamline knowledge ingestion and normalization – CloudWatch robotically collects AWS vended logs throughout accounts and AWS Areas, integrating with AWS Organizations from AWS companies together with AWS CloudTrail, Amazon Digital Non-public Cloud (Amazon VPC) Move Logs, AWS WAF entry logs, Amazon Route 53 resolver logs, and pre-built connectors for third-party sources reminiscent of endpoint (CrowdStrike, SentinelOne), id (Okta, Entra ID), cloud safety (Wiz), community safety (Zscaler, Palo Alto Networks), productiveness and collaboration (Microsoft Workplace 365, Home windows Occasion Logs, and GitHub), together with IT service supervisor with ServiceNow CMDB. To normalize and course of your knowledge as they’re being ingested, CloudWatch affords managed OCSF conversion for varied AWS and third-party knowledge sources and different processors reminiscent of Grok for customized parsing, field-level operations, and string manipulations.
- Scale back expensive log knowledge administration – CloudWatch consolidates log administration right into a single service with built-in governance capabilities with out storing and sustaining a number of copies of the identical knowledge throughout completely different instruments and knowledge shops. The unified knowledge retailer of CloudWatch eliminates the necessity for advanced ETL pipelines and reduces your operational prices and administration overhead wanted to keep up a number of separate knowledge shops and instruments.
- Uncover enterprise insights from log knowledge – You’ll be able to run queries in CloudWatch utilizing pure language queries and standard question languages reminiscent of LogsQL, PPL, and SQL by means of a single interface, or question your knowledge utilizing your most well-liked analytics instruments by means of Apache Iceberg-compatible tables. The brand new Sides interface offers you intuitive filtering by supply, utility, account, area, and log sort, which you need to use to run queries throughout log teams of a number of AWS accounts and Areas with clever parameter inference.
Within the subsequent sections we discover the brand new log administration and analytics options of the CloudWatch Logs!
1. Information discovery and administration by knowledge sources and kinds
You’ll be able to see a high-level overview of logs and all knowledge sources with a brand new Logs Administration View within the CloudWatch console. To get began, go to the CloudWatch console and select Log Administration beneath the Logs menu within the left navigation pane. Within the Abstract tab, you possibly can observe your logs knowledge sources and kinds, insights into how your log teams are doing throughout ingestion, and anomalies.
Select the Information sources tab to search out and handle your log knowledge by knowledge sources, varieties, and fields. CloudWatch ingests and robotically categorizes knowledge sources by AWS companies, third-party, or customized sources reminiscent of utility logs.
Select the Information supply actions to combine S3 Tables to make future logs for chosen knowledge sources. You will have the flexibleness to investigate the logs by means of Athena and Amazon Redshift and different question engines reminiscent of Spark utilizing Iceberg appropriate entry patterns. With this integration, logs from CloudWatch can be found in a read-only aws-cloudwatch S3 Tables bucket.
Once you select a particular knowledge supply reminiscent of CloudTrail knowledge, you possibly can view the main points of the info supply that features data relating to knowledge format, pipeline, sides/subject indexes, S3 Tables affiliation, and the variety of logs with that knowledge supply. You’ll be able to observe all log teams included on this knowledge supply and kind and edit a supply/sort subject index coverage utilizing the brand new schema assist.
To study extra about handle your knowledge sources and index coverage, go to Information sources within the Amazon CloudWatch Logs Consumer Information.
2. Ingestion and transformation utilizing CloudWatch pipelines
You’ll be able to create pipelines to streamline amassing, reworking, and routing telemetry and safety knowledge whereas standardizing knowledge codecs to optimize observability and safety knowledge administration. The brand new pipeline characteristic of CloudWatch connects knowledge from a listing of information sources, with the intention to add and configure pipeline processors from a library to parse, enrich, and standardize knowledge.
Within the Pipeline tab, select Add pipeline. It reveals you the pipeline configuration wizard. This wizard guides you thru 5 steps the place you possibly can select the info supply and different supply particulars reminiscent of log supply varieties, configure vacation spot, configure as much as 19 processors to carry out an motion in your knowledge (reminiscent of filtering, reworking, or enriching), and at last evaluation and deploy the pipeline.
You even have the choice to create pipelines by means of the brand new Ingestion expertise in CloudWatch. To study extra about arrange and handle the pipelines, go to Pipelines within the Amazon CloudWatch Logs Consumer Information.
3. Enhanced analytics and querying based mostly on knowledge sources
You’ll be able to improve analytics with assist for Sides and querying based mostly on knowledge sources. Sides allow interactive exploration and drill-down into logs and their values are robotically extracted based mostly on the chosen time interval.
Select the Sides tab within the Log Insights beneath the Logs menu within the left navigation pane. You’ll be able to view out there sides and values that seem within the panel. Select a number of sides and values to interactively discover your knowledge. I select Sides relating to a VPC Move Logs group and motion, question to listing the 5 most frequent patterns in my VPC Move Logs by means of the AI question generator, and get the end result patterns.
It can save you your question with the chosen Sides and values that you’ve got specified. Once you subsequent select your saved question, the logs to be queried have the pre-specified sides and values. To study extra about Side administration, go to Sides within the CloudWatch Logs Consumer Information.
As I beforehand famous, you possibly can combine knowledge sources into S3 Tables and question collectively. For instance, utilizing a Question Editor in Athena, you possibly can question correlates community site visitors with AWS API exercise from a particular IP vary (174.163.137.*) by becoming a member of VPC Move Logs with CloudTrail logs based mostly on matching supply IP addresses.
The sort of built-in search is especially beneficial for safety monitoring, incident investigation, and suspicious conduct detection. You’ll be able to view if an IP that’s making community connections can be performing delicate AWS operations reminiscent of creating customers, modifying safety teams, or accessing knowledge.
To study extra, go to S3 Tables integration with CloudWatch within the CloudWatch Logs Consumer Information.
Now out there
New log administration options of Amazon CloudWatch can be found right this moment in all AWS Areas besides the AWS GovCloud (US) Areas and China Areas. For Regional availability and future roadmap, go to the AWS Capabilities by Area. There aren’t any upfront commitments or minimal charges, and also you pay for the utilization of present CloudWatch Logs for knowledge ingestion, storage, and queries. To study extra, go to the CloudWatch pricing web page.
Give it a strive within the CloudWatch console. To study extra, go to the CloudWatch product web page and ship suggestions to AWS re:Publish for CloudWatch Logs or by means of your traditional AWS Assist contacts.
— Channy
