Logging mechanism
InfoScale runs primarily as daemonsets on an OpenShift or a Kubernetes cluster. To access logs of a failed pod/container, the logs must persist beyond the lifecycle of the container. Containerized InfoScale generates logs as log files and container logs. Logs are helpful for debugging purposes. Log files generated by containerized InfoScale persist on the hostPath
/var/VRTS/log of each host. You can access Container logs of running InfoScale pods/containers by using oc logs or kubectl logs command.
If DR Controller is configured, controller logs are also included in the Container logs. DR controller logs are independently generated on all peer clusters added in the Global Cluster Membership and hence, logs from all peer OpenShift or Kubernetes clusters must be collected.
To persist container logs beyond the lifecycle of the Container, following methods can be used.
EFK (ElasticSearch, Fluentd, Kibana) \- EFK is the default logging stack on OpenShift. See the OpenShift documentation for configuring EFK.
Fluentd log collector \- You can use Fluentd log collector to save the container logs at
/var/VRTS/log. Fluentd runs as a daemonset on the OpenShift or Kubernetes cluster. Hence, it can collect logs generated at each node. On OpenShift or Kubernetes, Fluentd needs to run with privileged mode to access hostPath volumes.
Run the following command to enable this:
oc adm policy add-scc-to-user privileged -z fluentd
Create a file
fluentd-infoscale-spec.yaml, and apply the following configuration by using oc command:
apiVersion: v1
kind: ConfigMap
metadata:
name: infoscale-fluentd-config
namespace: kube-system
data:
fluent.conf: |
<source>
@type tail
@id container-input
read_from_head true
format none
path "/var/log/containers/infoscale*.log"
pos_file "/var/log/infoscale.log.pos"
tag infoscale.tail.*
</source>
<match infoscale.tail.**>
@type file
format json
append true
path /containerlogs/${tag[5]}
<buffer tag>
flush_interval 5s
</buffer>
<format>
@type single_value
message_key message
</format>
</match>
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluentd
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-system
labels:
k8s-app: fluentd-logging
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
name: fluentd
template:
metadata:
labels:
k8s-app: fluentd-logging
kubernetes.io/cluster-service: "true"
name: fluentd
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
containers:
- name: fluentd
\#
\# On Openshift, fluentd needs to run as privileged due to hostPath mounts
\#
securityContext:
privileged: true
image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
env:
- name: FLUENT_UID
value: "0"
- name: FLUENT_ELASTICSEARCH_SED_DISABLE
value: "true"
volumeMounts:
- name: config-volume
mountPath: /fluentd/etc/fluent.conf
subPath: fluent.conf
- name: container-logs
mountPath: /var/log
- name: hostpath-containerlogs
mountPath: /containerlogs
\#
\# for docker containers (usual on vanilla kubernetes),enable below mountpoint
\# - name: varlibdockercontainers
\# mountPath: /var/lib/docker/containers
volumes:
- name: config-volume
configMap:
name: infoscale-fluentd-config
- name: container-logs
hostPath:
path: /var/log
type: Directory
- name: hostpath-containerlogs
hostPath:
path: /var/VRTS/log/containers
type: DirectoryOrCreate
\#
\# for docker containers (usual on vanilla kubernetes), enable below volume
\# - name: varlibdockercontainers
\# hostPath:
\# path: /var/lib/docker/containers
\# type: Directory
If you have configured DR, add the following to
fluentd-infoscale-spec.yaml. You can add it at the end of this file.
<source>
@type tail
@id container-input
read_from_head true
format none
path "/var/log/containers/infoscale-dr-manager*.log"
pos_file "/var/log/dr-controller.log.pos"
tag infoscale.tail.*
</source>
This configuration enables Fluentd to log all Infoscale containers to the hostPath directory
/var/VRTS/log/containers of each host.
Related information