Deploying TOBS (The Observability Stack) on LKE

Traducciones al Español
Estamos traduciendo nuestros guías y tutoriales al Español. Es posible que usted esté viendo una traducción generada automáticamente. Estamos trabajando con traductores profesionales para verificar las traducciones de nuestro sitio web. Este proyecto es un trabajo en curso.
Create a Linode account to try this guide with a $ credit.
This credit will be applied to any valid services used during your first  days.

TOBS, short for The Observability Stack, is a pre-packaged distribution of monitoring tools and dashboard interfaces. It can be installed on any existing Kubernetes cluster. It includes many of the most popular open-source observability tools such as Prometheus, Grafana, Promlens, TimescaleDB, and others. Together, these provide a maintainable solution to analyze the traffic on the server and identify any potential problems with a deployment. This guide covers deploying TOBS on LKE (Linode Kubernetes Engine) using Helm and the kubectl port-forward command for local access to your monitoring interfaces.

TOBS includes the following components:

  • OpenTelemetry collector is deployed to collect traces.
  • Alertmanager, is deployed alongside Prometheus, forms the alerting layer of the stack, and handles alerts generated by Prometheus.
  • Grafana is a data visualization and analytics tool that allows you to build dashboards and graphs for your metrics data.
  • PromLens helps users build PromQL queries with ease. PromLens is a PromQL query builder that helps you build, understand, and fix your queries much more effectively.
  • TimescaleDB is for long-term storage of metric data. Long-term storage provides the ability to perform post-hoc analysis on metric data over long periods of time. Such data analysis can be used for capacity planning, identifying slow-moving regressions, trend analysis, auditing, and more. For information about connecting to the database from the cluster, see TimescaleDB Documentation
  • Promscale provides the translation layer between Prometheus and the database. It allows the Prometheus server to store and retrieve metrics from TimescaleDB, and allows users to use PromQL on Promscale and Prometheus.
  • Prometheus is an open-source systems monitoring and altering stack. It has become the de-facto standard in metric monitoring and is the basis of standards such as OpenMetrics. It allows you to monitor and understand how your infrastructure and applications are performing. Service discovery allows Prometheus to automagically discover components within your Kubernetes cluster that are already emitting metrics.
  • kube-state-metrics exports the metrics related to Kubernetes resources such as the status and count of Kubernetes resources, with visibility of the desired resources and the current resources, as well as the trends in your cluster.
  • Node-Exporter is deployed to export node related metrics such as CPU, memory usage, and others from the Kubernetes cluster.

Before You Begin

Note
This guide was written using Kubernetes version 1.23.
  1. Deploy an LKE Cluster. This guide was written using an example node pool with three 4 GB Shared CPU Compute Instances. Depending on the workloads you plan to deploy on your cluster, you may consider using other plans with more available resources.

  2. Install Helm 3 to your local environment.

  3. Install kubectl to your local environment and connect to your cluster.

  4. Create the monitoring namespace on your LKE cluster:

    kubectl create namespace monitoring
  5. Add the stable Helm charts repository to your Helm repos:

    helm repo add stable https://charts.helm.sh/stable
  6. Update your Helm repositories:

    helm repo update

TOBS Minimal Deployment

In this section, learn to deploy TOBS for individual/local access with kubectl Port-Forward.

Deploy The Observability Stack

  1. Install a certificate manager for your LKE cluster:

    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
  2. Using Helm, deploy the TOBS release labeled lke-monitor in the monitoring namespace on your LKE cluster:

    helm repo add timescale https://charts.timescale.com/
    helm repo update
    helm install --wait lke-monitor timescale/tobs --namespace monitoring
  3. Verify that the Prometheus Operator has been deployed to your LKE cluster and its components are running and ready by checking the pods in the monitoring namespace:

    kubectl -n monitoring get pods

    You should see a similar output to the following:

    NAME                                                        READY   STATUS      RESTARTS      AGE
    alertmanager-tobs-kube-prometheus-alertmanager-0            2/2     Running     0             2m13s
    lke-monitor-connection-secret-j4sdh                         0/1     Completed   0             2m35s
    lke-monitor-grafana-54d979dcf5-tkkgj                        3/3     Running     2 (65s ago)   2m32s
    lke-monitor-grafana-db-swm8g                                0/1     Completed   3             2m35s
    lke-monitor-kube-state-metrics-6bc5c44b9-g8r5g              1/1     Running     0             2m27s
    lke-monitor-prometheus-node-exporter-b4vvg                  1/1     Running     0             2m33s
    lke-monitor-prometheus-node-exporter-bbcnd                  1/1     Running     0             2m34s
    lke-monitor-prometheus-node-exporter-frrfp                  1/1     Running     0             2m26s
    lke-monitor-promlens-569cfbd586-bkhrr                       1/1     Running     0             2m34s
    lke-monitor-promscale-86d574986c-9wj2z                      1/1     Running     4 (64s ago)   2m27s
    lke-monitor-timescaledb-0                                   1/1     Running     0             2m30s
    opentelemetry-operator-controller-manager-8cf5c85c8-krdj5   2/2     Running     0             2m27s
    prometheus-tobs-kube-prometheus-prometheus-0                2/2     Running     0             2m13s
    tobs-kube-prometheus-operator-5b4f674986-55r4k              1/1     Running     0             2m34s

Access Monitoring Interfaces with Port-Forward

  1. List the services running in the monitoring namespace and review their respective ports:

    kubectl -n monitoring get svc

    You should see an output similar to the following:

    NAME                                                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                           AGE
    alertmanager-operated                                       ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   3m41s
    lke-monitor                                                 ClusterIP   10.128.40.142    <none>        5432/TCP                     4m3s
    lke-monitor-config                                          ClusterIP   None             <none>        8008/TCP                     4m3s
    lke-monitor-grafana                                         ClusterIP   10.128.102.243   <none>        80/TCP                       4m3s
    lke-monitor-kube-state-metrics                              ClusterIP   10.128.208.39    <none>        8080/TCP                     4m3s
    lke-monitor-prometheus-node-exporter                        ClusterIP   10.128.170.88    <none>        9100/TCP                     4m3s
    lke-monitor-promlens                                        ClusterIP   10.128.45.92     <none>        80/TCP                       4m3s
    lke-monitor-promscale-connector                             ClusterIP   10.128.198.88    <none>        9201/TCP,9202/TCP            4m3s
    lke-monitor-replica                                         ClusterIP   10.128.137.189   <none>        5432/TCP                     4m3s
    opentelemetry-operator-controller-manager-metrics-service   ClusterIP   10.128.45.42     <none>        8443/TCP                     4m3s
    opentelemetry-operator-webhook-service                      ClusterIP   10.128.12.89     <none>        443/TCP                      4m3s
    prometheus-operated                                         ClusterIP   None             <none>        9090/TCP                     3m41s
    lke-monitor-kube-prometheus-alertmanager                           ClusterIP   10.128.33.44     <none>        9093/TCP                     4m3s
    lke-monitor-kube-prometheus-operator                               ClusterIP   10.128.175.39    <none>        443/TCP                      4m3s
    lke-monitor-kube-prometheus-prometheus                             ClusterIP   10.128.106.173   <none>        9090/TCP                     4m3s

    From the above output, the resource services you will access have the corresponding ports:

    ResourceService NamePort
    Prometheuslke-monitor-kube-prometheus-prometheus9090
    Alertmanagerlke-monitor-kube-prometheus-alertmanager9093
    Grafanalke-monitor-grafana80
  2. Use kubectl port-forward to open a connection to a service, then access the service’s interface by entering the corresponding address in your web browser:

    Note
    Press control+C on your keyboard to terminate a port-forward process after entering any of the following commands.
    • To provide access to the Prometheus interface at the address 127.0.0.1:9090 in your web browser, enter:

      kubectl -n monitoring \
      port-forward \
      svc/lke-monitor-kube-prometheus-prometheus \
      9090
    • To provide access to the Alertmanager interface at the address 127.0.0.1:9093 in your web browser, enter:

      kubectl -n monitoring \
      port-forward \
      svc/lke-monitor-kube-prometheus-alertmanager  \
      9093
    • To provide access to the Grafana interface at the address 127.0.0.1:8081 in your web browser, enter:

      kubectl -n monitoring \
      port-forward \
      svc/lke-monitor-grafana  \
      8081:80

      When accessing the Grafana interface, log in as admin. You can get the password using:

      kubectl get secret --namespace monitoring lke-monitor-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

      The Grafana dashboards are accessible at Dashboards > Manage from the left navigation bar.

TOBS eliminates the need to maintain configuration details for each of the applications, while providing standardized monitoring for the applications running on your cluster.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

This page was originally published on


Your Feedback Is Important

Let us know if this guide was helpful to you.


Join the conversation.
Read other comments or post your own below. Comments must be respectful, constructive, and relevant to the topic of the guide. Do not post external links or advertisements. Before posting, consider if your comment would be better addressed by contacting our Support team or asking on our Community Site.
The Disqus commenting system for Linode Docs requires the acceptance of Functional Cookies, which allow us to analyze site usage so we can measure and improve performance. To view and create comments for this article, please update your Cookie Preferences on this website and refresh this web page. Please note: You must have JavaScript enabled in your browser.