diff --git a/.devcontainer/devcontainer.json.tpl b/.devcontainer/devcontainer.json.tpl
index f4297af..c0b5dec 100644
--- a/.devcontainer/devcontainer.json.tpl
+++ b/.devcontainer/devcontainer.json.tpl
@@ -8,9 +8,9 @@
// Use ubuntu-22.04 or ubuntu-18.04 on local arm64/Apple Silicon.
"args": { "VARIANT": "ubuntu-22.04" }
},
- "mounts": [
- "source=/Users/prabhatsharma/.aws,target=/home/vscode/.aws,type=bind,consistency=cached"
- ],
+ // "mounts": [
+ // "source=/Users/prabhatsharma/.aws,target=/home/vscode/.aws,type=bind,consistency=cached"
+ // ],
// Set *default* container specific settings.json values on container create.
"settings": {},
diff --git a/docs/.pages b/docs/.pages
index fcfb59f..bb9021b 100644
--- a/docs/.pages
+++ b/docs/.pages
@@ -1,12 +1,12 @@
nav:
-- Introduction: index.md
+- Overview: overview
+- Features: features
- Getting started: getting-started.md
- Quickstart: quickstart.md
- Enterprise Edition Installation Guide: openobserve-enterprise-edition-installation-guide.md
- Releases: releases.md
- Example Queries: example-queries.md
- SQL reference: sql_reference.md
-- Architecture: architecture.md
- HA Deployment: ha_deployment.md
- Environment Variables: environment-variables.md
- Storage: storage.md
diff --git a/docs/features/.pages b/docs/features/.pages
new file mode 100644
index 0000000..ad9e121
--- /dev/null
+++ b/docs/features/.pages
@@ -0,0 +1,8 @@
+nav:
+ - Log Management: logs.md
+ - Metric Monitoring: metrics.md
+ - Distributed Tracing: distributed-tracing.md
+ - Frontent Observability: frontend.md
+ - Enterprise Features: enterprise.md
+ - What's next: upcoming.md
+
diff --git a/docs/features/distributed-tracing.md b/docs/features/distributed-tracing.md
new file mode 100644
index 0000000..3636f7b
--- /dev/null
+++ b/docs/features/distributed-tracing.md
@@ -0,0 +1,73 @@
+# Distributed Tracing
+
+OpenObserve provides powerful distributed tracing capabilities that enable you to track requests as they flow through your microservices architecture, identify performance bottlenecks, and troubleshoot complex distributed systems with ease.
+
+## Overview
+
+Distributed tracing in OpenObserve allows you to visualize the complete journey of requests across multiple services, understand service dependencies, and pinpoint the root cause of performance issues. Built for modern cloud-native applications, OpenObserve efficiently handles trace ingestion, correlation, and analysis at enterprise scale while maintaining exceptional query performance.
+
+
+*Distributed Tracing overview page*
+
+## Key Features
+
+### Trace Ingestion
+- **OpenTelemetry Native**: Full support for OpenTelemetry protocol with automatic trace collection
+- **Multiple Protocols**: Compatible with Jaeger, Zipkin, and custom tracing formats
+- **High Throughput**: Process millions of spans per second with sub-millisecond latency
+- **Sampling Strategies**: Intelligent sampling to balance observability with performance impact
+- **Batch Processing**: Optimized batch ingestion for maximum efficiency
+
+### Trace Structure & Data
+- **Hierarchical Spans**: Complete trace trees showing parent-child relationships between operations
+- **Rich Metadata**: Capture detailed span attributes, tags, and contextual information
+- **Error Tracking**: Automatic error detection and classification within traces
+- **Service Mapping**: Dynamic service topology discovery and visualization
+- **Custom Instrumentation**: Support for manual and automatic instrumentation
+
+### Trace Analysis & Search
+
+- **Advanced Filtering**: Search traces by service, operation, duration, errors, and custom attributes
+
+
+
+- **Dependency Analysis**: Visualize service dependencies and communication patterns
+
+- **Performance Profiling**: Identify slow operations and bottlenecks across your distributed system
+
+- **Error Investigation**: Quickly locate and analyze failed requests and exceptions
+
+### Visualization & Insights
+
+- **Gantt Chart Views**: Interactive timeline visualization showing span relationships and durations
+
+- **Service Map**: Real-time service topology with performance metrics and error rates
+
+- **Flame Graphs**: Detailed performance analysis with hierarchical span breakdown
+
+- **Trace Comparison**: Side-by-side comparison of traces for performance analysis
+
+### Correlation & Context
+
+- **Metrics Correlation**: Link traces to related metrics and logs for complete observability
+
+- **Log Integration**: Automatic correlation between trace spans and application logs
+
+- **Alert Integration**: Trigger alerts based on trace patterns, error rates, and performance thresholds
+
+### Storage & Performance
+
+- **Efficient Compression**: Advanced compression techniques reduce storage overhead by up to 85%
+
+- **Smart Indexing**: High-performance indexing optimized for trace queries and span searches
+
+- **Tail-based Sampling**: Intelligent sampling decisions based on complete trace context
+
+- **Retention Management**: [Flexible retention policies](../user-guide/streams/extended-retention.md) for cost-effective long-term storage
+
+### Integration & Standards
+- **OpenTelemetry Ecosystem**: Full compatibility with OpenTelemetry collectors and instrumentation libraries
+- **Jaeger Integration**: Seamless migration from Jaeger with native protocol support
+- **Zipkin Compatibility**: Support for Zipkin trace format and existing instrumentation
+- **Cloud Native**: Native Kubernetes integration with automatic service discovery
+- **API Access**: Comprehensive REST APIs for programmatic trace access and analysis
diff --git a/docs/features/enterprise.md b/docs/features/enterprise.md
new file mode 100644
index 0000000..d4984a1
--- /dev/null
+++ b/docs/features/enterprise.md
@@ -0,0 +1,21 @@
+# Enterprise Features
+
+## Overview
+Enterprise tier includes all standard features plus the following enterprise-specific capabilities. Available free for up to 200 GB of ingestion per day.
+
+## Enterprise-Only Features
+
+### Data Management
+- **Extended Data Retention** - Keep data longer than standard retention periods
+- **Federated Search / Super Cluster** - Search across multiple clusters from single interface
+
+### Security & Access
+- **SSO (Single Sign On)** - Integrate with existing identity providers
+- **Role-Based Access Control (RBAC)** - Manage user permissions by role
+- **Sensitive Data Redaction** - Automatically mask sensitive information
+- **Cipher Keys** - Encryption and compliance support for HIPAA and PCI
+
+### Operations
+- **Query Management** - Optimize and manage query performance
+- **Workload Management (QoS)** - Prioritize and allocate resources
+- **Audit Trail** - Track all system activities and changes
\ No newline at end of file
diff --git a/docs/features/frontend.md b/docs/features/frontend.md
new file mode 100644
index 0000000..5dc435a
--- /dev/null
+++ b/docs/features/frontend.md
@@ -0,0 +1,111 @@
+# Frontend Observability
+
+OpenObserve provides comprehensive frontend observability capabilities that enable you to monitor user experiences, track performance metrics, and diagnose issues in web applications and mobile apps. Gain complete visibility into how users interact with your frontend applications and ensure optimal user experience across all devices and browsers.
+
+## Overview
+
+Frontend Observability in OpenObserve allows you to monitor real user experiences, track Core Web Vitals, capture JavaScript errors, and analyze user journeys across your web and mobile applications. Built for modern frontend architectures, OpenObserve efficiently collects, processes, and analyzes frontend telemetry data while maintaining minimal impact on application performance.
+
+## Key Features
+
+### Real User Monitoring (RUM)
+- **User Session Tracking**: Complete user session recording with interaction timelines
+- **Page Performance**: Detailed page load times, rendering metrics, and resource loading analysis
+- **Core Web Vitals**: Automatic tracking of LCP, FID, CLS, and other Google Core Web Vitals
+- **Geographic Insights**: User experience analysis by geographic location and network conditions
+- **Device & Browser Analytics**: Performance breakdowns by device type, browser, and OS
+
+### [Error Tracking & Debugging](../user-guide/rum.md#error-tracking)
+
+- **JavaScript Error Capture**: Automatic collection of JavaScript errors, exceptions, and promise rejections
+
+- **Source Map Integration**: Detailed stack traces with original source code for minified applications
+
+- **Error Context**: Rich error context including user actions, browser state, and session information
+
+- **Error Grouping**: Intelligent error grouping and deduplication for efficient issue management
+
+### [Performance Monitoring](../user-guide/rum.md#performance-monitoring)
+
+- **Page Load Analytics**: Comprehensive page load performance with waterfall charts and timing breakdowns
+
+- **Resource Monitoring**: Track loading times for images, scripts, stylesheets, and API calls
+
+- **Runtime Performance**: Monitor JavaScript execution times, memory usage, and CPU utilization
+
+- **Custom Performance Metrics**: Track application-specific performance indicators and business metrics
+
+### User Experience Analytics
+
+- **User Journey Mapping**: Visualize complete user paths through your application
+
+- **Interaction Tracking**: Monitor clicks, scrolls, form submissions, and custom user interactions
+
+- **Rage Click Detection**: Identify user frustration points with automatic rage click detection
+
+- **Conversion Funnel Analysis**: Track user conversion rates and identify drop-off points
+
+### Network & API Monitoring
+
+- **AJAX/Fetch Tracking**: Monitor all HTTP requests with response times, status codes, and payload sizes
+
+- **API Performance**: Detailed API call analysis with success rates and error patterns
+
+- **Network Conditions**: Track user network speed and connection quality impact on performance
+
+- **Third-Party Service Monitoring**: Monitor external service dependencies and their impact on user experience
+
+### Browser & Device Insights
+
+- **Browser Compatibility**: Identify browser-specific issues and compatibility problems
+
+- **Device Performance**: Analyze performance across different device capabilities and screen sizes
+
+- **Viewport Analytics**: Understanding of how users interact with different screen resolutions
+
+### [Session Replay & Recording](../user-guide/rum.md#session-replay)
+
+- **Session Recordings**: Visual playback of user sessions for detailed investigation
+
+- **Privacy Controls**: Configurable data masking and privacy protection for sensitive information
+
+- **Event Timeline**: Synchronized timeline of user actions, network requests, and application state
+
+- **Issue Correlation**: Link session recordings to specific errors and performance issues
+
+
+### Mobile App Observability
+
+- **Crash Reporting**: Comprehensive crash reporting for iOS and Android applications
+
+- **App Performance**: Monitor app launch times, screen rendering, and memory usage
+
+- **User Engagement**: Track app usage patterns, feature adoption, and user retention
+
+- **Network Performance**: Monitor API calls and network conditions in mobile environments
+
+### Data Collection & Privacy
+
+- **Lightweight SDKs**: Minimal impact JavaScript and mobile SDKs with optimized performance
+
+- **Sampling Strategies**: Intelligent sampling to balance data quality with performance impact
+
+- **GDPR Compliance**: Built-in privacy controls and data anonymization features
+
+- **Custom Data Collection**: Flexible APIs for collecting custom frontend metrics and events
+
+## Integration
+
+### Web Applications
+- **JavaScript SDK**: Easy integration with vanilla JavaScript, React, Vue, Angular, and other frameworks
+- **NPM Package**: Simple installation via package managers with TypeScript support
+- **CDN Integration**: Quick setup with CDN-hosted SDK for immediate implementation
+- **Framework Plugins**: Native plugins for popular frameworks and build tools
+
+### CI/CD Integration
+- **Build Integration**: Integrate observability setup into your build and deployment pipelines
+- **Source Map Upload**: Automatic source map upload for enhanced error debugging
+- **Performance Budgets**: Set performance budgets and fail builds on regression
+- **Release Tracking**: Correlate frontend performance with deployments and releases
+
+
diff --git a/docs/features/logs.md b/docs/features/logs.md
new file mode 100644
index 0000000..2d97206
--- /dev/null
+++ b/docs/features/logs.md
@@ -0,0 +1,53 @@
+# Logs
+
+OpenObserve provides powerful log management capabilities for collecting, storing, and analyzing log data from your applications and infrastructure.
+
+## Overview
+
+Logs in OpenObserve offer comprehensive observability into your system's behavior, allowing you to track events, debug issues, and monitor application performance. Built with high performance and cost efficiency in mind, OpenObserve handles log ingestion and querying at scale.
+
+
+*Logs Page view*
+
+## Key Features
+
+### Log Ingestion
+- **Multiple Protocols**: Support for various log shipping protocols including HTTP, syslog, and popular log shippers
+- **Structured & Unstructured**: Handle both JSON structured logs and plain text logs
+- **Real-time Processing**: Immediate indexing and availability for search and analysis
+
+### Search & Query
+
+- **Field Extraction**: Automatic parsing and extraction of log fields.
+
+
+
+The [Schema Settings](../user-guide/streams/schema-settings.md) tab in the Stream Details panel allows you to inspect and manage the schema used to store and query ingested data.
+
+- **Full-text Search**: Powerful search capabilities across all log fields
+
+
+
+- **SQL Queries**: Use familiar SQL syntax for complex log analysis
+
+
+
+- **Time-based Filtering**: Efficient time range queries for targeted log exploration
+
+
+
+### Storage & Performance
+- **Compressed Storage**: Efficient compression reduces storage costs significantly
+
+
+
+- **Fast Retrieval**: Optimized indexing for quick log searches and aggregations
+
+
+
+Know more about [Streams](../user-guide/streams/streams-in-openobserve.md) and its [details](../user-guide/streams/stream-details.md#stream-details)
+
+- **Retention Policies**: [Configurable data retention](../user-guide/streams/extended-retention.md) to manage storage costs
+
+
+
diff --git a/docs/features/metrics.md b/docs/features/metrics.md
new file mode 100644
index 0000000..695f230
--- /dev/null
+++ b/docs/features/metrics.md
@@ -0,0 +1,70 @@
+# Metrics
+
+OpenObserve provides comprehensive metrics collection, storage, and visualization capabilities for monitoring your applications and infrastructure performance in real-time.
+
+## Overview
+
+Metrics in OpenObserve enable you to track key performance indicators, monitor system health, and gain insights into your application's behavior over time. Designed for high-throughput environments, OpenObserve efficiently handles metrics ingestion, storage, and querying at scale while maintaining cost-effectiveness.
+
+
+*Metrics Page view*
+
+## Key Features
+
+### Metrics Ingestion
+- **Multiple Formats**: Support for various metrics formats including Prometheus, InfluxDB, and JSON
+- **Push & Pull Models**: Flexible ingestion supporting both push-based and pull-based collection methods
+- **High Throughput**: Handle millions of metrics per second with minimal latency
+- **Batch Processing**: Efficient batch ingestion for optimal performance
+
+### Data Types & Structure
+- **Time Series Data**: Native support for time-series metrics with timestamp precision
+- **Multi-dimensional**: Handle metrics with multiple labels and dimensions for detailed analysis
+- **Aggregation Functions**: Built-in support for common aggregation functions (sum, avg, min, max, count)
+- **Custom Metrics**: Flexible schema for application-specific metrics and KPIs
+
+### Query & Analysis
+
+- **PromQL Support**: Full compatibility with Prometheus Query Language for familiar querying
+
+
+
+- **SQL Interface**: Use SQL syntax for complex metrics analysis and reporting
+
+- **Time Range Selection**: Flexible time range queries with support for relative and absolute time periods
+
+
+
+- **Mathematical Operations**: Perform calculations and transformations on metrics data
+
+### Visualization & Dashboards
+
+- **Real-time Charts**: Interactive time-series visualizations with multiple chart types
+
+
+
+- **Custom Dashboards**: Create comprehensive dashboards with multiple metrics panels
+
+- **Alerting Integration**: Set up alerts based on metrics thresholds and conditions
+
+
+
+### Storage & Performance
+
+- **Optimized Compression**: Advanced compression algorithms reduce storage costs by up to 90%
+
+
+
+- **Efficient Indexing**: High-performance indexing for fast query execution across large datasets
+
+- **Downsampling**: Automatic data [downsampling](../user-guide/metrics/downsampling-metrics.md) for long-term storage optimization
+
+- **Retention Policies**: [Configurable retention settings](../user-guide/streams/extended-retention.md) to balance storage costs and data availability
+
+
+
+### Integration & Compatibility
+- **Prometheus Compatible**: Full compatibility with Prometheus ecosystem and exporters
+- **Grafana Integration**: Native support for Grafana dashboards and visualization
+- **API Access**: RESTful APIs for programmatic access to metrics data
+- **Standard Exporters**: Support for popular metrics exporters (Node Exporter, cAdvisor, etc.)
diff --git a/docs/features/upcoming.md b/docs/features/upcoming.md
new file mode 100644
index 0000000..a57ae9e
--- /dev/null
+++ b/docs/features/upcoming.md
@@ -0,0 +1,9 @@
+# Product Roadmap
+
+Our vision is to build the most performant and cost-effective observability platform that scales from startup to enterprise. We're committed to delivering lightning-fast analytics across logs, metrics, traces, RUM, error tracking, and session replay while keeping your storage costs 140x lower than traditional solutions.
+
+Every feature we develop is driven by real user needs and our commitment to making observability simple, powerful, and accessible to teams everywhere.
+
+Explore our current roadmap and upcoming features **[here](https://github.com/openobserve/openobserve/milestones)**
+
+Have ideas that could make OpenObserve even better? We'd love to hear from you! Share your thoughts and feature requests in our **[GitHub Discussions](https://github.com/openobserve/openobserve/discussions)** - your input directly shapes what we build next 🚀
\ No newline at end of file
diff --git a/docs/images/features/data-compression.png b/docs/images/features/data-compression.png
new file mode 100644
index 0000000..abf2518
Binary files /dev/null and b/docs/images/features/data-compression.png differ
diff --git a/docs/images/features/data-retention.png b/docs/images/features/data-retention.png
new file mode 100644
index 0000000..1a06592
Binary files /dev/null and b/docs/images/features/data-retention.png differ
diff --git a/docs/images/features/full-text-search.png b/docs/images/features/full-text-search.png
new file mode 100644
index 0000000..ada1fc8
Binary files /dev/null and b/docs/images/features/full-text-search.png differ
diff --git a/docs/images/features/index.png b/docs/images/features/index.png
new file mode 100644
index 0000000..4c5b8d5
Binary files /dev/null and b/docs/images/features/index.png differ
diff --git a/docs/images/features/log-feilds-detection.png b/docs/images/features/log-feilds-detection.png
new file mode 100644
index 0000000..25ed246
Binary files /dev/null and b/docs/images/features/log-feilds-detection.png differ
diff --git a/docs/images/features/logs-page.png b/docs/images/features/logs-page.png
new file mode 100644
index 0000000..5ae93c4
Binary files /dev/null and b/docs/images/features/logs-page.png differ
diff --git a/docs/images/features/metrics-alerts.png b/docs/images/features/metrics-alerts.png
new file mode 100644
index 0000000..a559907
Binary files /dev/null and b/docs/images/features/metrics-alerts.png differ
diff --git a/docs/images/features/metrics-charts.png b/docs/images/features/metrics-charts.png
new file mode 100644
index 0000000..d0b4b3d
Binary files /dev/null and b/docs/images/features/metrics-charts.png differ
diff --git a/docs/images/features/metrics-compression.png b/docs/images/features/metrics-compression.png
new file mode 100644
index 0000000..40453a9
Binary files /dev/null and b/docs/images/features/metrics-compression.png differ
diff --git a/docs/images/features/metrics-page.png b/docs/images/features/metrics-page.png
new file mode 100644
index 0000000..a444017
Binary files /dev/null and b/docs/images/features/metrics-page.png differ
diff --git a/docs/images/features/metrics-retention.png b/docs/images/features/metrics-retention.png
new file mode 100644
index 0000000..39e71b1
Binary files /dev/null and b/docs/images/features/metrics-retention.png differ
diff --git a/docs/images/features/metrics-time-selection.png b/docs/images/features/metrics-time-selection.png
new file mode 100644
index 0000000..ca4088a
Binary files /dev/null and b/docs/images/features/metrics-time-selection.png differ
diff --git a/docs/images/features/promql-queries.png b/docs/images/features/promql-queries.png
new file mode 100644
index 0000000..14744cf
Binary files /dev/null and b/docs/images/features/promql-queries.png differ
diff --git a/docs/images/features/sql-based-log-search.png b/docs/images/features/sql-based-log-search.png
new file mode 100644
index 0000000..6786286
Binary files /dev/null and b/docs/images/features/sql-based-log-search.png differ
diff --git a/docs/images/features/time-selection.png b/docs/images/features/time-selection.png
new file mode 100644
index 0000000..aa7ada7
Binary files /dev/null and b/docs/images/features/time-selection.png differ
diff --git a/docs/images/features/trace-search.png b/docs/images/features/trace-search.png
new file mode 100644
index 0000000..568cb24
Binary files /dev/null and b/docs/images/features/trace-search.png differ
diff --git a/docs/images/features/tracing-page.png b/docs/images/features/tracing-page.png
new file mode 100644
index 0000000..40b2963
Binary files /dev/null and b/docs/images/features/tracing-page.png differ
diff --git a/docs/index.md b/docs/index.md
index 9f14a41..374ba60 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,119 +1,51 @@
# OpenObserve: Introduction
-`OpenObserve` is a cloud native observability platform (`Logs`, `Metrics`, `Traces`) that provides `~140x lower storage costs` (compared to Elasticsearch. YMMV. Could be higher or lower based on data entropy) for real life log data, significantly lower operational cost and ease of use. It can scale to petabytes of data, is highly performant and allows you to sleep better at night 😀. If you are looking for an observability tool for logs, metrics and traces, do evaluate OpenObserve and how its approach towards observability could help you build better software, save money on observability costs, and sleep better.
-
-
-## Project Status, Features and Roadmap
-
-Following is the list of available features and roadmap.
-
-| # | Feature | Status |
-|---|---------------------------------------------------------------|---------------------|
-| 1 | Log search | Available |
-| 2 | Highly compressed storage of data | Available |
-| 3 | Dynamic evolution of schema | Available |
-| 4 | Out of the box authentication | Available |
-| 5 | Support of S3, MinIO, GCS, Azure blob for data storage | Available |
-| 6 | Advanced GUI | Available |
-| 7 | SQL based query language | Available |
-| 8 | Support for very high cardinality data | Available |
-| 9 | Search-around logs data | Available |
-| 10 | User defined Ingest and Query functions (VRL based) | Available |
-| 11 | Multi-tenancy | Available |
-| 12 | Ingestion API compatibility with Elasticsearch | Available |
-| 13 | Search and aggregation API compatibility with Elasticsearch | [Through zPlane](zplane) |
-| 14 | Standard alerts (Based on logs) | Available |
-| 15 | Real time Alerts (Based on logs) | Available |
-| 16 | High Availability (HA) and clustering | Available |
-| 17 | Stateless nodes | Available |
-| 18 | Localization for multiple languages | Available |
-| 19 | Prebuilt binaries for multiple platforms | Available |
-| 20 | Prebuilt container images for multiple platforms | Available |
-| 21 | Prebuilt container images for with SIMD acceleration | Available |
-| 22 | SIMD support for vectorized processing (AVX512 and Neon) | Available |
-| 23 | Dashboards | Available |
-| 24 | Metrics | Available |
-| 25 | PromQL support for metrics | Available (97% PromQL compliant) |
-| 26 | Traces | Available |
-| 27 | Standard alerts (Based on metrics) | Available |
-| 28 | Real time Alerts (Based on metrics) | Available |
-| 29 | Template based alert target (Allows alerting to slack, teams and many more) | Available |
-| 30 | Send alerts to Prometheus alertmanager | Available |
-| 31 | Ingest AWS logs (cloudwatch, VPC flow logs, AWS WAF and more) using Kinesis firehose | Available |
-| 32 | Single Sign On(SSO) | Available (Enterprise) |
-| 33 | RBAC (Role Based Access Control) | Available (Enterprise) |
-| 34 | Front end - Performance analytics | Available |
-| 35 | Front end - Session Replay | Available |
-| 36 | Front end - Error tracking | Available |
-| 37 | Log patterns | To start |
-| 38 | Anomaly detection | To start |
-| 39 | Correlation between logs, metrics and traces | To start |
-| 40 | Dashboard migration from Splunk, Kibana and Grafana | beta. [https://dc.openobserve.ai](https://dc.openobserve.ai) [https://github.com/openobserve/dashboard_converter](https://github.com/openobserve/dashboard_converter) |
-
-
-Please raise any new feature requests via [github issue tracker](https://github.com/openobserve/openobserve/issues).
-
-You can use either the open source version or [OpenObserve Cloud](https://cloud.openobserve.ai). [OpenObserve Cloud](https://cloud.openobserve.ai) is built on top of open source OpenObserve but has minor differences to account for its SaaS nature. We will highlight the differences in the documentation whenever needed.
-
-## Guiding principles
-
-We want to build the best software in the observability category in the world, and we believe that the below principles will keep us aligned towards that:
-
-1. Day 1: It should be easy to setup and use
- 1. You should be able to install (for self hosted option) or sign up (for SaaS platform) in under 2 minutes.
- 1. You should be able to start ingesting data in under 2 minutes and start observing the behavior of your applications without any major configuration.
-2. Day 2: It should not be painful to keep the system up and running
- 1. Application should be stable and in the case of issues should be able to heal itself automatically.
- 1. Majority of the users should be able to start using the system efficiently with ZERO configuration.
- 1. Scaling up/down should be as easy as changing the number of nodes in an autoscaling group (in AWS) or changing the number of replicas (in k8s).
- 1. Majority of the folks should not need backups or should be able to do it without DBA level skills.
- 1. Fear of upgrades should not make you lose your sleep
-3. Features and Usability: It should have good features and functionality to do the job efficiently
- 1. System should be highly usable from the get go - providing excellent ROI on the invested time. A great UI and API are important to achieve it.
- 1. Logs themselves do not provide you visibility into your application. You need metrics and traces as well and the ability to correlate them.
-4. Cost: It should be cost effective
- 1. You should not have to mortgage your house or company assets in order to run the system either in self hosted mode (with or without licensing cost) or for SaaS platform.
-5. Learning curve: It should allow beginners to do a lot of tasks easily and advanced users should be able to use most of their existing skills
- 1. A user who has never used the system should be able to set up and use the system efficiently for basic needs or should be able to use existing skills for advanced purposes.
-6. Performance: It should be highly performant
- 1. System should be highly performant for most of the use cases in the real world.
- 1. Many a times performance requires a tradeoff. In situations of tradeoffs, it should be generally acceptable to the majority of the users for the use case with excellent tradeoff value in return.
-
-## How does OpenObserve compare to Elasticsearch
-
-Elasticsearch is a general purpose search engine which can be used for app search or log search. OpenObserve is built specifically for log search. If you are looking for a lightweight alternative to Elasticsearch then you should take a look at ZincSearch.
-
-OpenObserve provides ability to index data in multiple ways to make it faster yet keep storage size low. It uses a combination of partitioning, bloom filters, inverted indexes, caching and columnar storage to make search and aggregation queries faster. You can combine these to find the right balance between storage and performance. [Uber found 80% of queries in their production environment to be aggregation queries](https://www.uber.com/en-IN/blog/logging/) and columnar data storage of OpenObserve means that aggregation queries will typically be much faster than Elasticsearch.
-
-Below is the result when we sent real life log data from our kubernetes cluster to both Elasticsearch and OpenObserve using fluentbit. This only pertains to storage. Cost of EBS volume is [8 cents/GB/Month (GP3)](https://aws.amazon.com/ebs/pricing/), cost of s3 is [2.3 cents/GB/month](https://aws.amazon.com/s3/pricing/). In HA mode in Elasticsearch you generally have 1 primary node and 2 replicas. You don't need to replicate s3 for data durability/availability as [AWS redundantly stores your objects on multiple devices across a minimum of three Availability Zones (AZs) in an Amazon S3 Region](https://aws.amazon.com/s3/faqs/).
-
-
-
-OpenObserve offers significant advantage of 140x lower storage costs compared to Elasticsearch in the above scenario (YMMV, you could get higher or lower values based on entropy of data). That does not even consider additional unused EBS volume capacity that needs to be available in order to not run out of disk space and the effort that it requires to keep monitoring disk usage so it is not filled.
-
-Stateless node architecture allows OpenObserve to scale horizontally without worrying about data replication or corruption challenges.
-
-OpenObserve's lack of index mapping and associated challenges provides a hassle-free experience in managing clusters.
-
-You will typically see much lower operational effort and cost in managing OpenObserve clusters compared to Elasticsearch.
-
-The platform's built-in GUI eliminates the need for another component like Kibana, and has awesome performance, thanks to Rust, without the challenges of JVM.
-
-In contrast to Elasticsearch, which is a general-purpose search engine that doubles as an observability tool, OpenObserve was built from the ground up as an observability tool, with high focus on delivering exceptional observability.
-
-## Elasticsearch compatibility
-
-OpenObserve `_bulk` API endpoint is elasticsearch compatible and can be used by log forwarders like fluentbit, fluentd and vector. Filebeat is supported through zPlane.
-
-Search and aggregation API compatibility with Elasticsearch is provided through zPlane.
-
-zPlane is the enterprise product offered by ZincLabs that among other things provides Elasticsearch search and aggregation compatibility. Learn more about it at [zPlane docs](zplane)
-## Are there any benchmarks?
-OpenObserve is currently under heavy development with many changes still happening to the core engine. We will do benchmarking soon as we complete implementation of some of the items at hand.
-
-In the meanwhile, there are hundreds of production installations of OpenObserve globally at small, mid tier and very large scale being used by startups and enterprises alike. Many have reported that OpenObserve is highly performant. Some of them have replaced 5-7 node Elasticsearch clusters with a single node of OpenObserve.
-
-Here is a [case study of Jidu](https://openobserve.ai/blog/jidu-journey-to-100-tracing-fidelity) that increased their throughput and query performance by 10x and reduced their storage costs by 10x by switching from Elasticsearch to OpenObserve, ingesting 10TB of data everyday. Jidu is a large EV manufacturer in China.
+## What is OpenObserve?
+
+**OpenObserve**, also referred to as O2, is a cloud native observability platform that unifies **logs, metrics, and traces** into a single, powerful solution. Built from the ground up for modern cloud environments, OpenObserve delivers enterprise-grade observability at a fraction of the cost and complexity of traditional solutions.
+
+## Why Choose OpenObserve?
+
+### Dramatic Cost Reduction
+Experience up to **140x lower storage costs** compared to Elasticsearch while maintaining full functionality. Our innovative architecture ensures you get more observability for less budget.
+
+### Effortless Scale
+Built to handle **petabyte-scale data** with ease. Whether you're a growing startup or a large enterprise, OpenObserve scales seamlessly with your needs without the operational overhead.
+
+### Performance First
+Engineered for speed with SIMD acceleration and vectorized processing. Get faster queries, real-time insights, and better performance across all your observability data.
+
+### Unified Experience
+Stop juggling multiple tools. OpenObserve brings logs, metrics, traces, frontend monitoring, and alerting into one cohesive platform with a single pane of glass.
+
+## Who Should Use OpenObserve?
+
+OpenObserve is perfect for:
+- **Engineering teams** looking to reduce observability costs without sacrificing capabilities
+- **DevOps professionals** who need reliable, scalable monitoring at any scale
+- **Organizations** migrating from expensive legacy solutions like Elasticsearch
+- **Companies** requiring comprehensive observability across cloud-native applications
+
+## Key Advantages
+
+**Cost Effective**: Dramatically lower storage and operational costs
+**Cloud Native**: Built for modern cloud environments and containerized workloads
+**Easy to Deploy**: Get started quickly with minimal configuration
+**Highly Compatible**: Works with existing Prometheus, Elasticsearch tooling and workflows
+**Enterprise Ready**: SSO, RBAC, and compliance features available
+
+## Ready to Get Started?
+
+OpenObserve's architectural approach can transform how you handle observability - reducing costs while improving performance and ease of use.
+
+**Next Steps:**
+- Explore our comprehensive [Feature List](../features) to see all capabilities
+- Check out our [Quick Start Guide](/getting-started.md) to begin evaluation
+- Join our [Community](https://github.com/openobserve/openobserve/discussions) to connect with other users
+
+*Sleep better at night knowing your observability stack is both powerful and affordable* 😴
+
+
diff --git a/docs/overview/.pages b/docs/overview/.pages
new file mode 100644
index 0000000..fe7dc87
--- /dev/null
+++ b/docs/overview/.pages
@@ -0,0 +1,6 @@
+nav:
+ - Introduction: index.md
+ - Our Principle: guiding-principles.md
+ - Comparison with Alternatives: comparison.md
+ - Architecture: architecture.md
+
diff --git a/docs/architecture.md b/docs/overview/architecture.md
similarity index 97%
rename from docs/architecture.md
rename to docs/overview/architecture.md
index af5795d..d408f88 100644
--- a/docs/architecture.md
+++ b/docs/overview/architecture.md
@@ -5,11 +5,11 @@ OpenObserve can be run in single node or in HA mode in a cluster.
## Single Node
-Please refer to [quickstart](./quickstart.md) for single node deployments.
+Please refer to [quickstart](../quickstart.md) for single node deployments.
### SQLite and Local disk
-Use this mode for light usage and testing or if HA is not a requirement for you. (You could still ingest and search over 2 TB on a single machine per day. On a mac M2 in our tests, you can ingest at ~31 MB/Second or 1.8 GB/Min or 2.6 TB/Day with default configuration). This is the default mode for running OpenObserve. Check [Quickstart](./quickstart.md) to find various ways to get this setup done.
+Use this mode for light usage and testing or if HA is not a requirement for you. (You could still ingest and search over 2 TB on a single machine per day. On a mac M2 in our tests, you can ingest at ~31 MB/Second or 1.8 GB/Min or 2.6 TB/Day with default configuration). This is the default mode for running OpenObserve. Check [Quickstart](../quickstart.md) to find various ways to get this setup done.
@@ -19,7 +19,7 @@ Use this mode for light usage and testing or if HA is not a requirement for you.
## High Availability (HA) mode
-Local disk storage is not supported in HA mode. Please refer to [HA Deployment](./ha_deployment.md) for cluster mode deployment.
+Local disk storage is not supported in HA mode. Please refer to [HA Deployment](../ha_deployment.md) for cluster mode deployment.
@@ -48,7 +48,6 @@ By choosing to build the system this way we are able to build a much more cost e
Ingester is used to receive ingest request and convert data into parquet format and store it in object storage. They store data temporarily in WAL before transferring it to object storage.
The data ingestion flow is:
-
1. receive data from HTTP / gRPC API request.
diff --git a/docs/overview/comparison.md b/docs/overview/comparison.md
new file mode 100644
index 0000000..7fe7968
--- /dev/null
+++ b/docs/overview/comparison.md
@@ -0,0 +1,39 @@
+## How does OpenObserve compare to Elasticsearch
+
+Elasticsearch is a general purpose search engine which can be used for app search or log search. OpenObserve is built specifically for log search. If you are looking for a lightweight alternative to Elasticsearch then you should take a look at ZincSearch.
+
+#### Technical Advantages
+OpenObserve provides ability to index data in multiple ways to make it faster yet keep storage size low. It uses a combination of:
+
+- Partitioning
+- Bloom filters
+- Inverted indexes
+- Caching
+- Columnar storage
+
+[Uber found 80% of queries in their production environment to be aggregation queries](https://www.uber.com/en-IN/blog/logging/) and columnar data storage of OpenObserve means that aggregation queries will typically be much faster than Elasticsearch.
+
+#### Storage Cost Comparison
+
+Below is the result when we sent real life log data from our kubernetes cluster to both Elasticsearch and OpenObserve using fluentbit. This only pertains to storage. Cost of EBS volume is [8 cents/GB/Month (GP3)](https://aws.amazon.com/ebs/pricing/), cost of s3 is [2.3 cents/GB/month](https://aws.amazon.com/s3/pricing/). In HA mode in Elasticsearch you generally have 1 primary node and 2 replicas. You don't need to replicate s3 for data durability/availability as [AWS redundantly stores your objects on multiple devices across a minimum of three Availability Zones (AZs) in an Amazon S3 Region](https://aws.amazon.com/s3/faqs/).
+
+
+
+OpenObserve offers significant advantage of 140x lower storage costs compared to Elasticsearch in the above scenario. Your actual results may vary depending on how compressible your specific log data is. This doesn't even consider additional unused EBS volume capacity and monitoring overhead.
+
+#### Operational Advantages
+
+- **Stateless Architecture**: Scale horizontally without data replication or corruption challenges
+- **No Index Mapping**: Hassle-free cluster management without index mapping complexities
+- **Lower Operational Cost**: Significantly reduced effort in managing clusters
+- **Built-in GUI**: Eliminates need for additional components like Kibana
+- **Rust Performance**: Awesome performance without JVM challenges
+- **Purpose-Built**: Built from ground up as observability tool, not general-purpose search
+
+## Elasticsearch compatibility
+
+OpenObserve `_bulk` API endpoint is elasticsearch compatible and can be used by log forwarders like fluentbit, fluentd and vector. Filebeat is supported through zPlane.
+
+Search and aggregation API compatibility with Elasticsearch is provided through zPlane.
+
+zPlane is the enterprise product offered by ZincLabs that among other things provides Elasticsearch search and aggregation compatibility. Learn more about it at [zPlane docs](zplane)
\ No newline at end of file
diff --git a/docs/downloads.md b/docs/overview/downloads.md
similarity index 100%
rename from docs/downloads.md
rename to docs/overview/downloads.md
diff --git a/docs/overview/guiding-principles.md b/docs/overview/guiding-principles.md
new file mode 100644
index 0000000..e1e9821
--- /dev/null
+++ b/docs/overview/guiding-principles.md
@@ -0,0 +1,56 @@
+# Our Principles
+
+We are building the best software in the observability category in the world. These principles guide every decision we make:
+
+## Instant Value
+**Get started in minutes, not hours**
+
+- Install or sign up in under 2 minutes
+- Start ingesting data and gaining insights immediately
+- Zero configuration required for common use cases
+- Immediate ROI on your time investment
+
+## Effortless Operations
+**Systems that manage themselves**
+
+- Self-healing and automatic recovery from issues
+- Simple scaling - as simple as adding nodes in in an autoscaling group (in AWS) or changing the number of replicas (in k8s).
+- No specialized database administration skills required
+- Worry-free upgrades and maintenance
+- Built-in reliability and stability
+
+## Complete Observability
+**Everything you need in one platform**
+
+- Unified logs, metrics, and traces with correlation
+- Intuitive UI and powerful APIs
+- Feature-rich yet simple to use
+- Designed for real-world workflows
+
+## Accessible Costs
+**Enterprise capabilities without enterprise pricing**
+
+- Dramatically lower storage and operational costs
+- Transparent pricing for both self-hosted and SaaS
+- No surprise bills or cost spirals
+- Optimize your observability budget
+
+## Universal Usability
+**Built for everyone**
+
+- Beginners can accomplish complex tasks easily
+- Experts can leverage existing skills and knowledge
+- Familiar interfaces and standard protocols
+- Gentle learning curve with powerful advanced features
+
+## Real-World Performance
+**Optimized for actual usage patterns**
+
+- High performance for common observability workloads
+- Smart tradeoffs that benefit the majority of users
+- Hardware-accelerated processing where it matters
+- Scales efficiently with your data growth
+
+---
+
+*These principles aren't just words - they're measurable commitments that drive our product development and user experience.*
\ No newline at end of file
diff --git a/docs/overview/index.md b/docs/overview/index.md
new file mode 100644
index 0000000..415d24e
--- /dev/null
+++ b/docs/overview/index.md
@@ -0,0 +1,51 @@
+# OpenObserve: Introduction
+
+## What is OpenObserve?
+
+**OpenObserve**, also referred to as O2, is a cloud native observability platform that unifies **logs, metrics, and traces** into a single, powerful solution. Built from the ground up for modern cloud environments, OpenObserve delivers enterprise-grade observability at a fraction of the cost and complexity of traditional solutions.
+
+## Why Choose OpenObserve?
+
+### Dramatic Cost Reduction
+Experience up to **140x lower storage costs** compared to Elasticsearch while maintaining full functionality. Our innovative architecture ensures you get more observability for less budget.
+
+### Effortless Scale
+Built to handle **petabyte-scale data** with ease. Whether you're a growing startup or a large enterprise, OpenObserve scales seamlessly with your needs without the operational overhead.
+
+### Performance First
+Engineered for speed with SIMD acceleration and vectorized processing. Get faster queries, real-time insights, and better performance across all your observability data.
+
+### Unified Experience
+Stop juggling multiple tools. OpenObserve brings logs, metrics, traces, frontend monitoring, and alerting into one cohesive platform with a single pane of glass.
+
+## Who Should Use OpenObserve?
+
+OpenObserve is perfect for:
+- **Engineering teams** looking to reduce observability costs without sacrificing capabilities
+- **DevOps professionals** who need reliable, scalable monitoring at any scale
+- **Organizations** migrating from expensive legacy solutions like Elasticsearch
+- **Companies** requiring comprehensive observability across cloud-native applications
+
+## Key Advantages
+
+**Cost Effective**: Dramatically lower storage and operational costs
+**Cloud Native**: Built for modern cloud environments and containerized workloads
+**Easy to Deploy**: Get started quickly with minimal configuration
+**Highly Compatible**: Works with existing Prometheus, Elasticsearch tooling and workflows
+**Enterprise Ready**: SSO, RBAC, and compliance features available
+
+## Ready to Get Started?
+
+OpenObserve's architectural approach can transform how you handle observability - reducing costs while improving performance and ease of use.
+
+**Next Steps:**
+- Explore our comprehensive [Feature List](../features/logs.md) to see all capabilities
+- Check out our [Quick Start Guide](../getting-started.md) to begin evaluation
+- Join our [Community](https://github.com/openobserve/openobserve/discussions) to connect with other users
+
+*Sleep better at night knowing your observability stack is both powerful and affordable*
+
+
+
+
+
diff --git a/overrides/partials/header.html b/overrides/partials/header.html
index 6f62732..a1c339a 100644
--- a/overrides/partials/header.html
+++ b/overrides/partials/header.html
@@ -42,7 +42,7 @@
{% if config.theme.palette %}
{% if not config.theme.palette is mapping %}
- {% include "partials/palette.html" %}
+ {# {% include "partials/palette.html" %} #}
{% endif %}
{% endif %}
{% if not config.theme.palette is mapping %}