Network Flow Log
This document was translated by GPT-4
Without needing to place any code into an application, DeepFlow automatically generates network flow logs for all services.
Database table name: flow_log.l4_flow_log.
# 1. Tags
Here is the list of automatically injected Tags: IP, protocol, port, network packet header fields, data collection location, cloud resources, K8s resources, K8s custom Labels. The detailed descriptions of these fields are as follows.
| Name | DisplayName | Description |
|---|---|---|
| _id | UID | |
| time | Time | Round end_time to seconds. |
| region | Region | |
| az | Availability Zone | |
| host | VM Hypervisor | Host running virtual machine. |
| chost | Cloud Host | Including virtual machines |
| vpc | VPC | |
| l2_vpc | Forwarding VPC | VPC where the MAC address is located. |
| subnet | Subnet | |
| router | Router | |
| dhcpgw | DHCP Gateway | |
| lb | Load Balancer | |
| lb_listener | Load Balancer Listener | |
| natgw | NAT Gateway | |
| redis | Redis | |
| rds | RDS | |
| pod_cluster | K8s Cluster | |
| pod_ns | K8s Namespace | |
| pod_node | K8s Node | |
| pod_ingress | K8s Ingress | |
| pod_service | K8s Service | |
| pod_group_type | K8s Workload Type | |
| pod_group | K8s Workload | Such as Deployment |
| pod | K8s POD | |
| service | Service | |
| resource_gl0_type | Type - K8s POD First | Deprecated，please use auto_instance_type. |
| resource_gl0 | Instance - K8s POD First | Deprecated，please use auto_instance. |
| resource_gl1_type | Type - K8s Workload First | Deprecated，please use auto_service_type. |
| resource_gl1 | Instance - K8s Workload First | Deprecated，please use auto_service. |
| resource_gl2_type | Type - K8s Service First | Deprecated，please use auto_service_type. |
| resource_gl2 | Instance - K8s Service First | Deprecated，please use auto_service. |
| auto_instance_type | Type - K8s POD First | The type of 'auto_instance'. |
| auto_instance | Instance - K8s POD First | The instance of IP |
| auto_service_type | Type - K8s Service First | The type of 'auto_service'. |
| auto_service | Instance - K8s Service First | On the basis of 'auto_instance' |
| gprocess | Process | |
| tap_port_host | Tap Port Host | Deprecated，please use capture_nic_host. |
| tap_port_chost | Tap Port Cloud Host | Deprecated，please use capture_nic_chost. |
| tap_port_pod_node | Tap Port K8s Node | Deprecated，please use capture_nic_pod_node. |
| capture_nic_host | Host of Capture NIC | |
| capture_nic_chost | Cloud Host of Capture NIC | |
| capture_nic_pod_node | K8s Node of Capture NIC | |
| host_ip | VM Hypervisor | The management IP address of VM Hypervisor. |
| host_hostname | VM Hypervisor | The hostname of VM Hypervisor. |
| chost_ip | Cloud Host | The primary IP address of Cloud Host. |
| chost_hostname | Cloud Host | The hostname of Cloud Host. |
| pod_node_ip | K8s Node | The primary IP address of K8s Node. |
| pod_node_hostname | K8s Node | The hostname of K8s Node. |
| k8s.label | K8s Label | |
| k8s.annotation | K8s Annotation | |
| k8s.env | K8s Env | |
| cloud.tag | Cloud Tag | |
| os.app | OS APP | |
| eth_type | Ether Type | |
| vlan | VLAN TAG | |
| mac | MAC Address | |
| ip | IP Address | |
| is_ipv4 | IPv4 Flag | |
| is_internet | Internet IP Flag | Whether the IP address is an external Internet address. |
| province | Province | The province to which the Internet IP address belongs. |
| protocol | Network Protocol | |
| tunnel_tier | Tunnel Tiers | |
| tunnel_type | Tunnel Type | |
| tunnel_tx_id | TX Tunnel ID | |
| tunnel_rx_id | RX Tunnel ID | |
| tunnel_tx_ip | TX Tunnel IP Address | |
| tunnel_tx_ip_0 | TX Tunnel src IP Address | |
| tunnel_tx_ip_1 | TX Tunnel dst IP Address | |
| tunnel_rx_ip | RX Tunnel IP Address | |
| tunnel_rx_ip_0 | RX Tunnel src IP Address | |
| tunnel_rx_ip_1 | RX Tunnel dst IP Address | |
| tunnel_tx_mac | TX Tunnel MAC Address | |
| tunnel_tx_mac_0 | TX Tunnel src MAC Address | |
| tunnel_tx_mac_1 | TX Tunnel dst MAC Address | |
| tunnel_rx_mac | RX Tunnel MAC Address | |
| tunnel_rx_mac_0 | RX Tunnel src MAC Address | |
| tunnel_rx_mac_1 | RX Tunnel dst MAC Address | |
| client_port | Client Port | |
| server_port | Server Port | |
| tcp_flags_bit | TCP Flag Set | The set of TCP flags in all packets in the current natural minute. |
| syn_seq | Seq no. of SYN Packet | |
| syn_ack_seq | Seq no. of SYN-ACK Packet | |
| last_keepalive_seq | Seq no. of Heartbeat Packet | Seq number in the most recent heartbeat packet. |
| last_keepalive_ack | Ack no. of Heartbeat Packet | Ack number in the most recent heartbeat packet. |
| l7_protocol | Application Protocol | |
| request_domain | Request Domain | |
| flow_id | Flow ID | |
| start_time | Start Time | Unit: microseconds. Indicates the start time of the flow within the current natural minute |
| end_time | End Time | Unit: microseconds. Indicates the end time of the flow within the current natural minute. If the flow is closed within this minute |
| close_type | Flow Close Type | |
| status | Status | Determined by the close_type and protocol: Normal/ForceReport/Non-TCP timeout/Disconnected* = Normal |
| is_new_flow | New Flow Flag | |
| signal_source | Signal Source | |
| tap | Traffic Access Point | Deprecated，please use capture_network_type. |
| capture_network_type | Network Location | The network location for capturing traffic uses a fixed value (Cloud Network) to represent intra-cloud traffic |
| vtap | DeepFlow Agent | Deprecated，please use agent. |
| agent | DeepFlow Agent | |
| nat_source | NAT Source | |
| tap_port | TAP Port Identifier | Deprecated |
| tap_port_name | TAP Port Name | Deprecated |
| tap_port_type | TAP Port Type | Deprecated |
| capture_nic | Capture NIC ID | When the value of tap_port_type is 'Local NIC' |
| capture_nic_name | Capture NIC Name | When the value of tap_port_type is 'Local NIC' |
| capture_nic_type | Capture NIC Type | Indicates the type of traffic collection location |
| tap_side | TAP Side | Deprecated |
| observation_point | Observation Point | The logical location of the collection location in the traffic path |
| l2_end | Boundary of L2 Network | Indicates whether the traffic is collected on the client NIC or the server NIC. |
| l3_end | Boundary of L3 Network | Indicates whether the traffic is collected in the Layer 2 network where the client or server is located. |
| has_pcap | PCAP File | Whether the PCAP file is stored |
| nat_real_ip | NAT IP Address | The real IP address before (after) NAT |
| nat_real_port | NAT Port | The real port number before NAT works |
generate from csv file: l4_flow_log.en
# 2. Metrics
Metrics list: throughput, load, latency, TCP exceptions, retransmissions, zero window. The detailed descriptions of these fields are as follows.
| Field | DisplayName | Unit | Description |
|---|---|---|---|
| byte | Byte | Byte | |
| byte_tx | Byte TX | Byte | |
| byte_rx | Byte RX | Byte | |
| total_byte_tx | Total Byte TX | Byte | |
| total_byte_rx | Total Byte RX | Byte | |
| packet | Packet | Packet | |
| packet_tx | Packet TX | Packet | |
| packet_rx | Packet RX | Packet | |
| total_packet_tx | Total Packet TX | Packet | |
| total_packet_rx | Total Packet RX | Packet | |
| l3_byte | L3 Payload | Byte | |
| l3_byte_tx | L3 Payload TX | Byte | |
| l3_byte_rx | L3 Payload RX | Byte | |
| bpp | Bytes per Packet | Byte | |
| bpp_tx | Bytes per Packet TX | Byte | |
| bpp_rx | Bytes per Packet RX | Byte | |
| new_flow | New Flow | Flow | |
| closed_flow | Closed Flow | Flow | |
| syn_count | SYN Packet | Packet | |
| synack_count | SYN-ACK Packet | Packet | |
| l4_byte | L4 Payload | Byte | |
| l4_byte_tx | L4 Payload TX | Byte | |
| l4_byte_rx | L4 Payload RX | Byte | |
| direction_score | Direction Score | The higher the score | |
| log_count | Log Count | ||
| retrans_syn | SYN Retransmission | Packet | |
| retrans_synack | SYN-ACK Retransmission | Packet | |
| retrans | TCP Retransmission | Packet | |
| retrans_tx | TCP Client Retransmission | Packet | |
| retrans_rx | TCP Server Retransmission | Packet | |
| zero_win | TCP ZeroWindow | Packet | |
| zero_win_tx | TCP Client ZeroWindow | Packet | |
| zero_win_rx | TCP Server ZeroWindow | Packet | |
| retrans_syn_ratio | SYN Retrans. % | % | |
| retrans_synack_ratio | SYN-ACK Retrans. % | % | |
| retrans_ratio | TCP Retrans. % | % | |
| retrans_tx_ratio | TCP Client Retrans. % | % | |
| retrans_rx_ratio | TCP Server Retrans. % | % | |
| zero_win_ratio | TCP ZeroWindow % | % | |
| zero_win_tx_ratio | TCP Client ZeroWindow % | % | |
| zero_win_rx_ratio | TCP Server ZeroWindow % | % | |
| tcp_establish_fail | Error | Flow | |
| client_establish_fail | Client Error | Flow | |
| server_establish_fail | Server Error | Flow | |
| tcp_establish_fail_ratio | Error % | % | |
| client_establish_fail_ratio | Client Error % | % | |
| server_establish_fail_ratio | Client Error % | % | |
| tcp_transfer_fail | Transfer Error | Flow | All transfer errors. |
| tcp_transfer_fail_ratio | Transfer Error % | % | |
| tcp_rst_fail | RST | Flow | All RST errors. |
| tcp_rst_fail_ratio | RST % | % | |
| client_source_port_reuse | Est. - Client Port Reuse | Flow | |
| server_syn_miss | Est. - Server SYN Miss | Flow | |
| client_establish_other_rst | Est. - Client Other RST | Flow | |
| client_ack_miss | Est. - Client ACK Miss | Flow | |
| server_reset | Est. - Server Direct RST | Flow | |
| server_establish_other_rst | Est. - Server Other RST | Flow | |
| client_rst_flow | Transfer - Client RST | Flow | |
| server_rst_flow | Transfer - Server RST | Flow | |
| server_queue_lack | Transfer - Server Queue Overflow | Flow | |
| tcp_timeout | Transfer - TCP Timeout | Flow | |
| client_half_close_flow | Close - Client Half Close | Flow | |
| server_half_close_flow | Close - Server Half Close | Flow | |
| rtt | Avg TCP Est. Delay | us | |
| tls_rtt | Avg TLS Est. Delay | us | |
| rtt_client | Avg TCP Est. Client Delay | us | |
| rtt_server | Avg TCP Est. Server Delay | us | |
| srt | Avg TCP/ICMP ACK Delay | us | |
| art | Avg Data Delay | us | |
| cit | Avg Client Idle Delay | us | |
| rtt_max | Max TCP Est. Delay | us | |
| tls_rtt_max | Max TLS Est. Delay | us | |
| rtt_client_max | Max TCP Est. Client Delay | us | |
| rtt_server_max | Max TCP Est. Server Delay | us | |
| srt_max | Max TCP/ICMP ACK Delay | us | |
| art_max | Max Data Delay | us | |
| cit_max | Max Client Idle Delay | us | |
| srt_sum | Total TCP/ICMP ACK Delay | us | |
| srt_count | TCP TCP/ICMP Delay Count | ||
| art_sum | Total Data Delay | us | |
| art_count | Data Delay Count | ||
| cit_sum | Total Client Idle Delay | us | |
| cit_count | Client Idele Delay Count | ||
| duration | Duration | us | The duration from start_time to the last packet (not end_time). |
| l7_request | Request | ||
| l7_response | Response | ||
| rrt | Avg App. Delay | us | |
| rrt_sum | Total App. Delay | us | |
| rrt_count | App. Delay Count | ||
| rrt_max | Max App. Delay | us | |
| l7_error | App. Error | ||
| l7_client_error | App. Client Error | ||
| l7_server_error | App. Server Error | ||
| l7_server_timeout | App. Server Timeout | ||
| l7_error_ratio | App. Error % | % | |
| l7_client_error_ratio | App. Client Error % | % | |
| l7_server_error_ratio | App. Server Error % | % | |
| l7_parse_failed | L7 Protocol Parse Failed | Packet | Cumulative number of application protocol parsing failures |
| row | Row Count |
generate from csv file: l4_flow_log.en
# 3. Grafana Dashboard
Based on the mentioned data, a rich Dashboard can be built via Grafana. We have predefined a Network - Flow Log Dashboard in Grafana. The result is as follows:
Network Flow Log
You can also visit DeepFlow Online Demo (opens new window) to check the effect.