• Best Practice

Special Environment Deployment

This document was translated by GPT-4

# 1. Special CNI

# 1.1 MACVlan

When K8s uses the macvlan CNI, only one virtual network card shared by all PODs can be seen under rootns, at this time, extra configuration is needed for the deepflow-agent:

  1. Create the agent-group and agent-group-config:

    deepflow-ctl agent-group create macvlan
deepflow-ctl agent-group-config create macvlan
    1
    2

  2. Obtain the macvlan agent-group ID:

    deepflow-ctl agent-group list  | grep macvlan
    1

  3. Create a new agent-group-config configuration file macvlan-agent-group-config.yaml:

    vtap_group_id: g-xxxxxx
## Regular Expression for TAP (Traffic Access Point)
## Length: [0, 65535]
## Default:
##   Localhost:   lo
##   Common NIC:  eth.*|en[osipx].*
##   QEMU VM NIC: tap.*
##   Flannel:     veth.*
##   Calico:      cali.*
##   Cilium:      lxc.*
##   Kube-OVN:    [0-9a-f]+_h$
## Note: Regular expression of NIC name for collecting traffic
tap_interface_regex: eth0
## Traffic Tap Mode
## Default: 0, means local.
## Options: 0, 1 (virtual mirror), 2 (physical mirror, aka. analyzer mode)
## Note: Mirror mode is used when deepflow-agent cannot directly capture the
##   traffic from the source. For example:
##   - in the K8s macvlan environment, capture the Pod traffic through the Node NIC
##   - in the Hyper-V environment, capture the VM traffic through the Hypervisor NIC
##   - in the ESXi environment, capture traffic through VDS/VSS local SPAN
##   - in the DPDK environment, capture traffic through DPDK ring buffer
##   Use Analyzer mode when deepflow-agent captures traffic through physical switch
##   mirroring.
tap_mode: 1
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25

  4. Create an agent-group-config:

    deepflow-ctl agent-group-config create -f macvlan-agent-group-config.yaml
    1

  5. Modify the agent-group of deepflow-agent:

    kubectl edit cm -n deepflow deepflow-agent
    1

    Add configuration:

    vtap-group-id-request: g-xxxxx
    1

    Stop deepflow-agent:

    kubectl -n deepflow  patch daemonset deepflow-agent  -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'
    1

    Using deepflow-ctl delete the macvlan's agent:

    deepflow-ctl agent delete <agent name>
    1

    Start deepflow-agent:

    kubectl -n deepflow  patch daemonset deepflow-agent --type json -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]'
    1

    Check the deepflow agent list, ensure the agent has joined the macvlan group:

    deepflow-ctl agent list
    1

# 2. Special K8s Resources or CRD

For these scenarios, you need to:

  • Enable or disable the corresponding resources in the advanced configuration of the Agent
  • Configure Kubernetes API permissions

# 2.1 OpenShift

In this scenario, we need disable the default Ingress resource acquisition and enable Route resource acquisition.

Agent advanced configuration are as follows:

static_config:
  kubernetes-resources:
    - name: ingresses
      disabled: true
    - name: routes
1
2
3
4
5

Need to add in the ClusterRole configuration:

rules:
  - apiGroups:
      - route.openshift.io
    resources:
      - routes
    verbs:
      - get
      - list
      - watch
1
2
3
4
5
6
7
8
9

# 2.2 OpenKruise

In this scenario, you need to get CloneSet and apps.kruise.io/StatefulSet resources from the API.

Agent advanced configuration are as follows:

static_config:
  kubernetes-resources:
    - name: clonesets
      group: apps.kruise.io
    - name: statefulsets
      group: apps
    - name: statefulsets
      group: apps.kruise.io
1
2
3
4
5
6
7
8

Here, you need to add Kubernetes apps/StatefulSet.

Need to add in the ClusterRole configuration:

- apiGroups:
    - apps.kruise.io
  resources:
    - clonesets
    - statefulsets
  verbs:
    - get
    - list
    - watch
1
2
3
4
5
6
7
8
9

# 3. Restricted Agent Runtime Permissions

# 3.1 Lack of Daemonset Deployment Permissions

When there is no permission to run the Daemonset in the Kubernetes cluster, but normal processes can be run directly on the K8s Node, this method can be used to implement Agent's deployment.

  • Deploy a deepflow-agent in a deployment manner
    • By setting the environment variable ONLY_WATCH_K8S_RESOURCE, this agent only implements functions of list-watching K8s resources and uploading to controller
    • All other functions of this agent will be automatically closed
    • When the agent requests the server, it informs that it is on watch-k8s, and the server updates this information to the MySQL database
    • This agent, which is used only for watching, will not appear on the Agent list
  • In this K8s cluster, a regular deepflow-agent runs in the form of a Linux process on each K8s Node
    • Since these agents do not have the IN_CONTAINER environment variable, they do not list-watch K8s resources
    • These agents continue to acquire POD's IP and MAC addresses and synchronize to the server
    • These agents will complete all the observation data collection functions
    • The agent type that the server delivers to these agents is K8s

# 3.1.1 Deploy DeepFlow Agent in Deployment Mode

cat << EOF > values-custom.yaml
deployComponent:
- "watcher"
clusterNAME: process-example
EOF

helm install deepflow -n deepflow deepflow/deepflow-agent --create-namespace \
  -f values-custom.yaml
1
2
3
4
5
6
7
8

After deployment, a Domain corresponding to this K8s cluster will be automatically created. By deepflow-ctl domain list to get the kubernetes-cluster-id of process-example cluster, then continue with the following operations.

# 3.1.2 Deploy DeepFlow Agent in Ordinary Process Mode

  • Refer to Deploy DeepFlow Agent on Traditional Servers, but there is no need to create Domain
  • Modify the agent configuration file /etc/deepflow-agent/deepflow-agent.yaml, fill in the cluster ID obtained in the previous step for kubernetes-cluster-id

# 3.2 No Request to Apiserver Allowed

By default, the DeepFlow Agent runs in Daemonset mode in K8s. But in some cases, to protect the apiserver from overload, the Daemonset is not allowed to request the apiserver. In this case, you can also use a similar method to the "No Daemonset deployment permission" in this article for deployment:

  • Deploy a deepflow-agent deployment, which only responsible for list-watch apiserver and sync K8s resource information
  • Deploy a deepflow-agent daemonset, any Pod won't list-watch apiserver