• Release Notes

v6.1 CE Release Notes

This document was translated by GPT-4

# 1. v6.1.8.5 [2023/01/12] LTS

  • Bugfix

# 2. v6.1.8.4 [2023/01/05]

# 2.1 New Features

  • Management
    • Support for specifying database names when using managed MySQL

# 3. v6.1.8.3 [2022/12/29]

  • Bugfix

# 4. v6.1.8.2 [2022/12/22]

# 4.1 New Features

  • AutoMetrics、AutoTracing
    • Added support for SOFARPC protocol
  • Grafana
    • Added support to be used as Grafana Tempo data source, displaying Tracing data on the Tempo page

# 5. v6.1.8.1 [2022/12/15]

# 5.1 New Features

# 6. v6.1.8 [2022/12/09]

# 6.1 New Features

  • AutoMetrics、AutoTracing
  • Management
    • Added a signal_source tag field to the data table to represent the signal source. The three currently supported signal sources are: Packet (cBPF), eBPF, OTel.
    • deepflow-agent supports running in K8s macvlan/ipvlan CNI environments and automatically calculates Pod relationships and performance indicators, with detailed configuration methods for macvlan environments referred to the documentation
    • Supports deepflow-agent running in K8s Pod without SYS_ADMIN permissions, detailed permissions requirements refer to documentation (opens new window)
    • Supports configuration of the maximum length (opens new window) of eBPF Socket Data collected by deepflow-agent to improve collection performance
    • Support configuration of resource information (opens new window) synchronized in deepflow-server to ingester module to reduce resource consumption of ingester module's AutoTagging function under multiple K8s clusters

# 6.2 Optimization

  • Management
    • By compressing and clipping messages, it reduces the memory overhead of deepflow-agent when watching K8s apiserver
    • Optimized the memory consumption of deepflow-server in calculating service Tag by aggregating Pod
    • By pre-aggregating memory, optimize the write pressure of deepflow-server on flow_tag database
    • Managed MySQL services support non-root user connections

# 7. v6.1.7 [2022/11/23]

# 7.1 New Features

# 7.2 Optimization

  • SQL API
    • When the automatic grouping resource type resource_glX_type is an IP address, it directly reuses resource_glX_id to represent the subnet ID and resource_glX to represent the IP address
  • Management
    • Lower the database permission requirements when using managed RDS and ClickHouse
    • Auto-balance the number of deepflow-agents for each deepflow-server service
    • Monitor the internal call chain of deepflow-server using OpenTelemetry

# 8. v6.1.6 [2022/11/09]

# 8.1 New Features

  • AutoMetrics
    • HTTP protocol adds attribute.http_user_agent and attribute.http_referer fields
    • Added attribute.rpc_service field, the value is ServiceName of Dubbo/gRPC
    • Added endpoint field, the value is ServiceName/MethodName of Dubbo/gRPC
    • Supports marking HTTP2 data compliant with gRPC protocol specification as gRPC (rather than HTTP2) protocol
  • AutoTagging
    • Support the synchronization of AWS public cloud resource information and automatically inject it into observability data as Tags
    • Support for tagging observability data of K8s clusters hosted on public clouds with cloud resources and container labels at the same time
  • Management
    • Supports configuration of deepflow-agent to turn on or off parsing of various application protocols
    • Supports configuration of the regular expression of Golang/openssl process name for deepflow-agent to collect data via eBPF uprobe
    • Supports deepflow-agent standalone mode, where Flow Log and Request Log are written to local log files
    • Supports i18n, with English as the default display

# 8.2 Optimization

  • AutoMetrics
    • Matches SQL keywords to reduce false positive rates of MySQL and PostgreSQL protocol recognition
  • Grafana
    • Displays SQL query statements in the Query Editor of Panel to help developers understand how to call API
    • Optimizes the display of empty field information in the Distributed Tracing flame graph
  • Management
    • Optimizes traffic between deepflow-server and clickhouse, giving priority to writing to the same node's clickhouse Pod
    • Supports the compression of OTel Span data received by deepflow-agent. The bandwidth consumption when sent to deepflow-server can be reduced by about 7 times

# 9. v6.1.5 [2022/10/26]

# 9.1 New Feature

  • AutoMetrics, AutoTracing
    • Supports the collection of PostgreSQL performance indicators and access logs, and associates them with distributed tracing
    • Supports the collection of performance indicators and access logs of HTTPS using the openssl library, and associates them with distributed tracing
  • Integration
    • deepflow-server supports providing a RemoteRead interface for Prometheus
    • deepflow-agent supports bypassing otel-collector to receive OpenTelemetry data directly
    • Grafana Variable's query statement supports using custom variables and built-in variables, please see the documentation for details. Use cases include:
      • Using the current selected value of the variable pod_cluster to filter the range of values for the variable pod
      • Use the input content of the variable ingress_wildcard to change the range of values of the variable ingress
      • Using the current value of the built-in variables $__from and $__to to enhance the query speed of the variable value range
      • Added two zero-intrusion observability Dashboards in Grafana: K8s Ingresss, SQL Monitoring
  • SQL API
    • Tag of type string_enum and int_enum supports using the Enum() function to translate Value to Name for query filtering and result return
    • Supports SELECT tags/attributes/labels to query all tag.X/attribute.X/label.X fields of each row of data, no need to specify specific field names
  • Management
    • Supports ClickHouse cold data using disk (as an alternative to object storage)
    • Support to use deepflow-ctl agent rebalance to balance deepflow-agent to added and restored deepflow-server

# 9.2 Optimization

  • AutoMetrics
  • AutoTagging
    • Added deepflow-ctl cloud info command to debug resource information synchronized from cloud platform API
  • SmartEncoding
    • Recycle the Tag encoding value of deleted resources, enhance the compression ratio and query speed
  • SQL API
    • Optimizes the display_name of the enumerated values of the server_port Tag field, includes the corresponding int value to avoid unclear meaning
  • Management
    • deepflow-server changed to use Deployment Controller for deployment, please upgrade the helm chart when upgrading
    • deepflow-agent supports running in non-privileged mode, please consult the specific permissions requirements from the documentation (opens new window)
    • Optimized the mapping relationship between Prometheus metrics and DeepFlow Table, each Metrics corresponds to a Table

# 10. v6.1.4 [2022/10/12]

# 10.1 New Features

  • AutoTagging
    • Supports the synchronization of Tencent public cloud resource information and automatically injects it into observability data as tags
  • AutoTracing
    • Supports the association of application Span and network Span in environments where eBPF cannot run, eliminating tracing blind spots
  • SQL API
    • The show tags command now returns fields of type map, such as labels, attributes, tags
    • The show tag values command now supports limit, offset, like parameters
  • Production Environment Deployment
    • Supports the use of managed ClickHouse and MySQL
    • deepflow-agent supports configuration distribution by deepflow-server when registration is not completed
  • Grafana
    • Added DeepFlow self-monitoring Dashboard

# 10.2 Optimization

  • SQL API
    • Metric data in OTel Span can be returned through the show metrics API
  • System Capability
    • Supports completion of deepflow-server master election without depending on sidecar
    • deepflow-server supports backward compatibility with deepflow-agent
    • Default sync with NTP server on the node where deepflow-server container is located
    • Standardized process -v output

# 11. v6.1.3 [2022/09/23]

# 11.1 Application

  • AutoMetrics
    • New metrics added: Client wait latency, SYN packet count, SYN-ACK packet count, SYN retransmission packet count, SYN-ACK retransmission packet count.
    • Support for collecting Golang HTTP2, HTTP2_TLS call logs using eBPF uprobe.
    • Supports collecting uprobe data from Golang processes that have trimmed the standard symbol table (Golang >= 1.13 and < 1.18).
  • AutoTracing
    • Supports associating eBPF uprobe Span with cBPF Span, OTel Span, and displaying them in the tracing flame graph.
  • AutoTagging
    • Automatically generates cloud server tags for K8s nodes not associated with a cloud server.
    • Supports syncing resource information from Huawei's public cloud.
  • Querier SQL API
    • Automatically returns fields after GROUP BY, no need to explicitly declare them after SELECT.
  • Grafana
    • Added thumbnail displays for Topology Diagrams (DeepFlow Topo) and Tracing Flame Graphs (DeepFlow AppTracing).
    • Optimized Span Tip in tracing flame graphs to show the proportion of time spent by the Span itself.

# 11.2 System

  • Reduced the wait time for agent to come online during the first deployment from 7 minutes to 4 minutes.
  • Access to deepflow-server and clickhouse within the same K8s cluster no longer uses NodeIP.
  • deepflow-server defaults to externalTrafficPolicy=Cluster to avoid issues with externalTrafficPolicy=Local in some environments and certain CNI compatibility issues, can be manually changed to Local for optimizing cross-cluster traffic.
  • deepflow-server adds ext-metrics-ttl, flow-metrics-ttl, flow-log-ttl configuration parameters to set data retention duration.
  • deepflow-agent supports writing l4_flow_log and l7_flow_log to local files.
  • deepflow-agent removes dependency on libbpf.

# 12. v6.1.2 [2022/09/08]

V6.1.2 is the second version after open source release, marking the start of a bi-weekly minor version release schedule.

# 12.1 Application

  • AutoMetrics
    • MySQL now includes parsing capability for COM_STMT_PREPARE, COM_STMT_EXECUTE, COM_STMT_FETCH, COM_STMT_CLOSE commands (opens new window).
    • Supports collection of MQTT 3.1 call logs and performance metrics.
  • SQL API
    • Supports SELECT labels to retrieve all custom label columns.

# 12.2 System

  • deepflow-agent
    • Added non-active IP metric data configuration item; when turned off, IPs without reply traffic will be aggregated.
    • Uses BPF pre-filtering for collection interfaces to enhance collection performance.
    • Provides deb installation package.
  • deepflow-ctl
    • agent-group-config update no longer requires specifying the agent-group-id parameter.

# 13. v6.1.1 [2022/08/25]

V6.1.1, developed over three months, is a major iterative version in the V6 series and the first community edition released by DeepFlow. From now on, enterprise and community editions will iterate in sync, with a new minor version released every two weeks, culminating in the final minor version of V6.1 (LTS version) after about four minor versions.

Aiming to create a highly automated open-source observability platform, this version involves significant software architecture adjustments, including merging too many microservices, eliminating some components' dependencies on deployment modes like DaemonSet and HostNet, and rewriting some Python modules in Golang.

Most importantly, this version greatly enhances application performance monitoring capabilities:

  • Supports integration and association of OpenTelemetry, SkyWalking tracing data, unlocking full-stack distributed tracing capabilities, and improving communication efficiency between different development teams.
  • Supports integration and association of Prometheus, Telegraf metric data, consolidating system, application, and business metrics on one platform, enabling collaboration among operations, development, and business teams.
  • Enhanced AutoTagging and SmartEncoding capabilities, supporting injection of more K8s custom Labels.
  • Enhanced eBPF AutoTracing capabilities, supporting collection of Golang HTTPS calls.

Note: The following Release Notes do not include enterprise version features.

# 13.1 Application Features

  • Metric Data Enhancement
    • Supports integration of Prometheus metric data (implementing the remote_write interface).
    • Supports integration of Telegraf metric data.
    • Supports automatic injection of cloud resources, container resources, and custom K8s Label tags into integrated metric data (AutoTagging).
  • Distributed Tracing Enhancement
    • Supports integration of OpenTelemetry tracing data.
    • Supports integration of SkyWalking tracing data.
    • Supports associating application, system, and network Spans, achieving full-stack distributed tracing capabilities.
    • Supports automatic injection of cloud resources, container resources, and custom K8s Label tags into integrated tracing data (AutoTagging).
  • Enhanced Application Call Collection Capability
    • Supports eBPF collection of Golang HTTPS call logs and performance metrics.
    • Supports eBPF/BPF collection of MQTT 3.1 call logs and performance metrics (Beta).
    • Removes port number restriction in application protocol parsing.
  • Enhanced Support for Grafana
    • Introduces an interactive DeepFlow Query Editor, a new zero-barrier query condition editor.
    • Adds DeepFlow AppTracing Panel for drawing call chain tracing flame graphs.
    • Adds DeepFlow Topo Panel for drawing call topology diagrams.
  • Improved Search Capabilities
    • Supports querying historical data of deleted resources.
    • Supports using custom K8s Labels to search all data.
    • Supports grouping all data based on custom K8s Labels.
    • Supports creating template variables based on all searchable fields.
  • Adjusted Flow Log Aggregation Logic
    • Flow logs only record TCP Flags that appear in the current cycle.
    • Delays timeout after collecting FIN-FIN-ACK sequence, waiting for the last ACK.
    • Stores the VLAN field in traffic data.
  • Added Math Operators for Metric Quantities.

# 13.2 System Features

  • Provides deepflow-ctl command-line tool for CLI management.
  • Agent (Collector)
    • Supports parsing multiple standards of TraceID, SpanID in both HTTP and Dubbo protocols.
    • Supports collecting network and application performance data on VIPs from loopback interfaces.
    • Default network interface collection supports Calico, Flannel, Qemu, Cilium, Kube-OVN, localhost, physical network cards.
    • Moves all configuration items that do not hinder the collector startup to be distributed by the controller, offering declarative group configuration APIs.
    • Supports remote limitation of CPU usage by deepflow-agent (Rust).
    • Supports self-healing after collector fails to allocate controller or data node.
  • Server (Controller, Data Node)
    • Supports synchronizing custom K8s Labels.
    • Supports delayed deletion of information on deleted resources.
    • Provides a unified Schema API and SQL Query API for all data tables.
    • Merges main modules of the controller and data node into deepflow-server.
    • Optimizes deployment methods of controller and data nodes, reducing dependence on DaemonSet and HostNet.
    • Optimizes the main controller election logic, eliminating dependence on Zookeeper.
    • Eliminates storing plain text secrets in the database.