docs: update elk stack
This commit is contained in:
committed by
Stefan Rotsch
parent
85bef49afe
commit
b69bc0eacd
@@ -2,13 +2,9 @@
|
||||
title: "Elasticsearch"
|
||||
ring: trial
|
||||
quadrant: platforms-and-aoe-services
|
||||
tags: [devops]
|
||||
---
|
||||
|
||||
Elasticsearch is a REST-based search and analytics engine based on Lucene.
|
||||
Unlike its competitor Apache Solr, it was developed in the beginning with clustering and scaling in mind.
|
||||
It allows you to create complex queries while still delivering results very fast.
|
||||
Elasticsearch is a REST-based search and analytics engine based on [Lucene](https://lucene.apache.org/). Unlike its competitor [Apache Solr](https://solr.apache.org/), it was developed from the beginning with clustering and scaling in mind. It allows you to create complex queries while still delivering results very quickly.
|
||||
|
||||
At AOE, we use Elasticsearch for logging as well as our own search solution [Searchperience®](http://www.searchperience.com/).
|
||||
We recently moved the Searchperience stack from Solr to Elasticsearch and think this was the right decision.
|
||||
Especially in terms of scaling, ease of use and performance, Elasticsearch really shines.
|
||||
Also, the API design took some learnings from Apache SOLR into account - for example, the queryDSL is a powerful way of describing different search use cases with highly flexible support of aggregations, etc.
|
||||
At AOE, we use Elasticsearch for logging as well as our own search solution, [Searchperience®](http://www.searchperience.com/). We recently moved the Searchperience stack from Solr to Elasticsearch and believe this was the right decision. Especially in terms of scaling, ease of use, and performance, Elasticsearch really shines. Additionally, the API design incorporates learnings from Apache Solr - for example, the queryDSL is a powerful way of describing different search use cases with highly flexible support for aggregations, etc.
|
||||
|
||||
@@ -2,20 +2,18 @@
|
||||
title: "ELK Stack"
|
||||
ring: adopt
|
||||
quadrant: platforms-and-aoe-services
|
||||
tags: [devops]
|
||||
---
|
||||
|
||||
The company behind Elasticsearch offers a very nice solution for logging and analysis of distributed data such as logfiles.
|
||||
The [ELK Stack](https://www.elastic.co/de/elastic-stack) is a powerful trio of tools that work together to provide a comprehensive solution for logging, searching, visualizing, and analyzing data in real time:
|
||||
|
||||
In today's increasingly distributed IT systems, it's very helpful to have a central view of what is going on in your systems - and of course nobody can and wants to look in different logfiles on different servers.
|
||||
A central logging solution provides the option to detect potential relationships between different events more easily.
|
||||
Also, also it can be used to extract useful KPIs or to visualize information on dashboards.
|
||||
- **[Elasticsearch](../platforms-and-aoe-services/elasticsearch.html)**: A search and analytics engine that indexes and stores data.
|
||||
- **Logstash**: A data processing pipeline that ingests and transforms various data formats.
|
||||
- **Kibana**: A visualization tool that creates interactive dashboards and reports.
|
||||
|
||||
The abbreviation "[ELK](https://www.elastic.co/products) Stack" stands for the Tools <u>E</u>lasticsearch, <u>L</u>ogstash and <u>K</u>ibana:
|
||||
Together, they provide a solution for collecting data the ability to search, visualize and analyze data in real time.
|
||||
In today's complex IT environments, having a centralized logging solution is essential. It allows you to:
|
||||
- Gain a unified view of system activities.
|
||||
- Detect correlations between different events more easily.
|
||||
- Extract valuable KPIs and visualize them on customizable dashboards.
|
||||
|
||||
Logstash is used to process and forward different data (or logfile) formats.
|
||||
Elasticsearch is used as a search index and together with the Kibana plugin you can configure highly individual dashboards.
|
||||
Recently, there are also the Beats Tools joining this toolstack to ship data to Elasticsearch.
|
||||
|
||||
We have been using the ELK Stack for several years now in several projects and different infrastructure setups - we use it to visualize traffic, certain KPIs or just to analyze and search in application logs.
|
||||
We encourage all teams to use such a solution and take care to write useful logs in your applications.
|
||||
At AOE, we have been using the ELK Stack for years in various projects and infrastructure setups. It has proven invaluable for visualizing traffic, tracking KPIs, and analyzing application logs. We highly recommend that all teams adopt this solution and focus on generating meaningful logs in their applications.
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
title: "Elasticsearch"
|
||||
ring: adopt
|
||||
quadrant: platforms-and-aoe-services
|
||||
tags: [devops]
|
||||
---
|
||||
|
||||
We are continuing to use Elasticsearch successfully in [Searchperience®] and have benefited from the aggregation features for related use cases such as rendering category trees.
|
||||
We are also using Elasticsearch for some microservices as our persistence solution.
|
||||
We are continuing to use Elasticsearch successfully in [Searchperience®](http://www.searchperience.com/) and have benefited from the aggregation features for related use cases such as rendering category trees. We also use Elasticsearch as a persistence solution for various microservices.
|
||||
|
||||
This is why we have updated its status to **adopt**.
|
||||
|
||||
@@ -2,5 +2,6 @@
|
||||
title: "ELK Stack"
|
||||
ring: adopt
|
||||
quadrant: platforms-and-aoe-services
|
||||
tags: [devops]
|
||||
featured: false
|
||||
---
|
||||
|
||||
@@ -2,14 +2,11 @@
|
||||
title: "Loki"
|
||||
ring: trial
|
||||
quadrant: tools
|
||||
featured: true
|
||||
tags: [devops]
|
||||
---
|
||||
|
||||
Archiving indexed log data with a system like Elasticsearch can be expensive and archiving it as simple text files makes it hard to query them.
|
||||
[Loki](https://grafana.com/oss/loki/) solves this issue by adding a reference database based on Kubernetes labels to each log line similar to Prometheus, but holding the log data inside a simple blob storage like S3.
|
||||
This allows the user to query the data by pre-defined labels and keeps the costs for indexing low.
|
||||
Archiving indexed log data with a system like [Elasticsearch](../platforms-and-aoe-services/elasticsearch.html) can be expensive, and archiving it as simple text files makes it hard to query. [Loki](https://grafana.com/oss/loki/) solves this issue by adding a reference database based on [Kubernetes](../platforms-and-aoe-services/kubernetes.html) labels to each log line, similar to [Prometheus](../platforms-and-aoe-services/prometheus.html), but storing the log data inside a simple blob storage like AWS S3. This allows the user to query the data by predefined labels and keeps the costs for indexing low.
|
||||
|
||||
Another benefit is the fact that does not have an endpoint for mutating log data which makes the data immutable from a potential compromised system.
|
||||
Another benefit is that Loki does not have an endpoint for mutating log data, making the data immutable from a potentially compromised system.
|
||||
|
||||
We at AOE are using it for longer term log archiving in several Kubernetes clusters.
|
||||
At AOE, we use Loki for long-term log archiving in several Kubernetes clusters.
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
---
|
||||
title: "Loki"
|
||||
ring: trial
|
||||
quadrant: platforms-and-aoe-services
|
||||
featured: true
|
||||
tags: [devops]
|
||||
---
|
||||
|
||||
Updated to adopt.
|
||||
@@ -5,9 +5,9 @@ quadrant: platforms-and-aoe-services
|
||||
tags: [devops]
|
||||
---
|
||||
|
||||
After having very positive experiences, we decided to replace our ELK stacks with Loki, primarily for the following reasons:
|
||||
After having very positive experiences, we decided to replace our [ELK stacks](../platforms-and-aoe-services/elk-stack.html) with Loki, primarily for the following reasons:
|
||||
|
||||
- Loki is significantly more cost-effective than the storage requirements of Elasticsearch used in the ELK stack.
|
||||
- Loki is significantly more cost-effective than the storage requirements of Elasticsearch.
|
||||
- The PromQL-like query language, familiar to users of Prometheus, makes it easier for DevOps and SRE teams who already use Prometheus for monitoring to work with logs.
|
||||
- Loki's native integration with Kubernetes simplifies the setup and configuration process.
|
||||
- Loki typically requires less maintenance and overhead compared to ELK.
|
||||
|
||||
9
radar/2024-06-01/elk-stack.md
Normal file
9
radar/2024-06-01/elk-stack.md
Normal file
@@ -0,0 +1,9 @@
|
||||
---
|
||||
title: "ELK Stack"
|
||||
ring: adopt
|
||||
quadrant: platforms-and-aoe-services
|
||||
tags: [devops]
|
||||
featured: false
|
||||
---
|
||||
|
||||
While we continue to recommend the ELK Stack for specific use cases, we now prefer [Loki](../platforms-and-aoe-services/loki.html) for most [Kubernetes](../platforms-and-aoe-services/kubernetes.html)-based setups due to its seamless integration, cost efficiency and user-friendly query language.
|
||||
Reference in New Issue
Block a user