Detailed Explanation of Distributed Storage Technology (4 Mainstream Storage Solutions)

HyperConverged

Distributed storage technology is the cornerstone of large-scale architectures. Below, I will focus on providing a detailed explanation of distributed storage technology

HDFS

HDFS (Hadoop Distributed File System) is the most commonly used distributed file system in the big data ecosystem. Its design goal is to achieve high throughput and suit scenarios involving large files and batch processing.

Core Components

It includes NameNode (for metadata management), DataNode (for data block storage), and Secondary/Backup NameNode.

HDFS Architecture

Client → HDFS NameNode, Secondary NameNode
Heartbeat, load balancing, replication, etc.
mikechen.cc
DataNode (multiple nodes) → Local Disk (for each DataNode)

Key Features

  • Optimized for large files and sequential read/write operations (suitable for MapReduce, Spark, etc.).
  • Centralized metadata management (via NameNode), enabling fast metadata querying. However, since NameNode is a critical point, HA (High Availability) configuration is required.
  • Adopts a default three-replica replication mechanism, ensuring fast recovery but incurring high space overhead; supports erasure coding based on HDFS (to save storage space).

Advantages and Disadvantages

  • Advantages: High throughput, mature ecosystem (deep integration with the Hadoop ecosystem), stability, and reliability.
  • Disadvantages: Not suitable for a large number of small files; average performance in low-latency random access; NameNode becomes a key concern in the architecture.

Applicable Scenarios

Offline big data processing, batch ETL (Extract, Transform, Load), massive log storage, and data lake scenarios.

Ceph

Ceph is a unified, distributed storage system designed to provide object storage, block storage, and file storage functions.

Architecture

Pool (multiple pools) → PG (Placement Group, multiple PGs under each pool)
RADOS Cluster
Host (multiple hosts) → Data
Object → OSD (Object Storage Daemon, multiple OSDs under each host)
File
Host (multiple hosts) → MON (Monitor), MGR (Manager)

Ceph provides RADOS (underlying distributed object storage), RBD (block device), CephFS (file system), and RGW (object gateway, compatible with S3/Swift protocols).

Key Features

  • Decentralized metadata (data location realized via the CRUSH algorithm, reducing single-point bottlenecks).
  • Supports two data protection strategies: replication and erasure coding, enabling flexible balancing of performance and storage space.
  • Integrates with CephFS, RBD, and RGW to provide a unified platform for different storage needs.

Advantages and Disadvantages

  • Advantages: High scalability, strong fault tolerance, comprehensive functions (unified platform for block/object/file storage).
  • Disadvantages: Complex deployment and tuning (requires fine-grained tuning of cluster parameters, network, and hardware); additional optimization may be needed for small-file performance and high-concurrency metadata operations.

Applicable Scenarios

Cloud platforms, virtualization backends (OpenStack), unified multi-type storage, large-scale object storage, and enterprise-level distributed storage.

GlusterFS

GlusterFS is an open-source distributed file system that aggregates multiple storage servers to form a unified namespace.

Key Features

  • Relatively simple deployment, supporting horizontal scaling and elastic expansion.
  • Data distribution and replication are controlled by the volume mechanism, supporting replication, striping, and distributed volume modes.
  • Can be directly mounted as a POSIX-compliant file system via FUSE, and supports protocols such as NFS/SMB.

Advantages and Disadvantages

  • Advantages: Easy to get started with, suitable for small-to-medium-scale distributed file requirements, and good protocol compatibility.
  • Disadvantages: May encounter performance bottlenecks under extremely large-scale or high-concurrency metadata operations; certain advanced functions require careful evaluation in terms of stability and performance.

Applicable Scenarios

File sharing services, media storage, enterprise internal file systems, and scenarios requiring protocol compatibility (POSIX/NFS).

FastDFS

FastDFS is an open-source, high-performance distributed file system specifically designed for massive small-file storage and high-concurrency access.

The core concept of FastDFS is lightweight and decentralization, with a very concise architecture. It mainly consists of two core roles:

Architecture

Client (Client 1, Client 2, ..., Client M)
Tracker Cluster (Tracker 1, Tracker 2, ..., Tracker N): Responsible for "scheduling center" functions, does not store any file data, only records and manages metadata of storage servers (e.g., which group a storage server belongs to, current status (online/offline), remaining space, etc.).
Storage Cluster: Serves as the "data warehouse", responsible for actual file storage, synchronization, and management. It is divided into multiple Groups (Group 1, Group 2, ..., Group K). Each Group contains multiple Storage Servers (e.g., Group 1 has Storage Server 11, Storage Server 12, ..., Storage Server 1X; Group 2 has Storage Server 21, Storage Server 22, ..., Storage Server 2Y; etc.).

Request/Response Flow: Client sends requests to Tracker Cluster → Tracker Cluster returns responses (e.g., storage server information) → Client interacts with the specified Storage Server in the Storage Cluster (sends file storage/retrieval requests and receives responses).

In summary, with its concise and efficient architecture, FastDFS has become a classic solution for many Internet applications to address massive file storage issues, especially performing excellently in small-file storage scenarios.