Customer Self-Service Portal

Forward logs from Google Cloud Platform to Site24x7 with Dataflow

GCP dataflow

Google Cloud Platform (GCP) enables organizations to create and scale applications. Activities in applications, whether on Compute Engine or other services from virtual machines to serverless environments on GCP, produce a significant amount of logs. Logs play a crucial role in helping you achieve effective observability and troubleshooting. But the logs may experience irregular surges in data ingestion during major system events, posing challenges for network overhead. 

This challenge is addressed by services like Dataflow, a serverless, fully managed solution that automatically takes care of resource provisioning, scaling, and maintenance without requiring infrastructure management. 

Google Cloud Dataflow offers a unified programming model that supports both real-time data streaming for immediate insights and batch processing for historical analysis—all with scalability. This flexibility ensures that the system can efficiently handle varying workloads. In contrast with the real-time model, where each log event is sent to your sink as an individual network request, resulting in high network overhead, batch processing in Dataflow involves batching and compressing log messages before transmitting them for processing.

Forward your logs from GCP to Site24x7

You can deploy the Site24x7 Dataflow Flex Template to GCP, creating a Dataflow job. The process of forwarding logs from GCP using the Dataflow Flex Template is as follows:

Log forwarding from GCP

Google Cloud services generate logs, which are then written to Google Cloud Logging. To manage and route these logs, a log routing sink is configured to publish log entries to a Google Cloud Pub/Sub topic. 

Subsequently, a Dataflow Flex Template is deployed to GCP. This template is designed to create a Dataflow job responsible for reading logs from the specified Pub/Sub topic and exporting the processed log data to Site24x7. This job reads logs from the topic created, automatically batches and compresses log messages, and exports processed log data to our log management system. This gives you scalable ingestion, thus reducing the data outflow cost. 

You can also collect Virtual Private Cloud flow logs from GCP and forward them to our log management system for improved monitoring, security, and analysis of your network traffic and cloud environment.

Forwarding GCP logs to our log management solution helps you visualize the logs, which along with customizable dashboards and reports provide a tailored view of your system's health. Our centralized logging feature provides a single console for managing logs seamlessly across GCP regions. With Site24x7's advanced alerting features, timely responses to critical events are ensured. Initiating IT Automation allows for the automatic resolution of application problems and eliminates manual intervention. 

You can follow the steps in this document to forward logs from GCP to Site24x7 via Dataflow for improved log management, monitoring, and troubleshooting.
​
You can also monitor all your Google Cloud services, pinpoint performance bottlenecks, and promptly resolve issues using Site24x7's GCP monitoring. Explore the comprehensive list of GCP services supported for monitoring here.