Skip to content

Commit

Permalink
Merge pull request #1394 from qdrant/feature-resharding
Browse files Browse the repository at this point in the history
[docs] Resharding
  • Loading branch information
timvisee authored Jan 17, 2025
2 parents ae53349 + 1bd2e74 commit ead3cd8
Show file tree
Hide file tree
Showing 2 changed files with 61 additions and 3 deletions.
52 changes: 50 additions & 2 deletions qdrant-landing/content/documentation/cloud/cluster-scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ Note, that it is not possible to scale down the disk space of the cluster due to

## Horizontal scaling

Vertical scaling can be an effective way to improve the performance of a cluster and extend the capacity, but it has some limitations. The main disadvantage of vertical scaling is that there are limits to how much a cluster can be expanded. At some point, adding more resources to a cluster can become impractical or cost-prohibitive.
Vertical scaling can be an effective way to improve the performance of a cluster and extend the capacity, but it has some limitations. The main disadvantage of vertical scaling is that there are limits to how much a cluster can be expanded. At some point, adding more resources to a cluster can become impractical or cost-prohibitive.

In such cases, horizontal scaling may be a more effective solution.
In such cases, horizontal scaling may be a more effective solution.

Horizontal scaling, also known as horizontal expansion, is the process of increasing the capacity of a cluster by adding more nodes and distributing the load and data among them. The horizontal scaling at Qdrant starts on the collection level. You have to choose the number of shards you want to distribute your collection around while creating the collection. Please refer to the [sharding documentation](/documentation/guides/distributed_deployment/#sharding) section for details.

Expand All @@ -38,3 +38,51 @@ When scaling down horizontally, the cloud platform will automatically ensure tha
We will be glad to consult you on an optimal strategy for scaling.

[Let us know](/documentation/support/) your needs and decide together on a proper solution.

## Resharding

*Available as of Qdrant v1.13.0*

<aside role="status">Resharding is exclusively available across our <a href="/documentation/cloud-intro/">cloud</a> offering, including <a href="/documentation/hybrid-cloud/">Hybrid</a> and <a href="/documentation/private-cloud/">Private</a> Cloud.</aside>

When creating a collection, it has a specific number of shards. The ideal number of shards might change as your cluster evolves.

Resharding allows you to change the number of shards in your existing collections, both up and down, without having to recreate the collection from scratch.

Resharding is a transparent process, meaning that the collection is still available while resharding is going on without having downtime. This allows you to scale from one node to any number of nodes and back, keeping your data perfectly distributed without compromise.

To increase the number of shards (reshard up), use the [Update collection cluster setup API](https://api.qdrant.tech/master/api-reference/distributed/update-collection-cluster) to initiate the resharding process:

```http
POST /collections/{collection_name}/cluster
{
"start_resharding": {
"direction": "up",
"shard_key": null
}
}
```

To decrease the number of shards (reshard down), you may specify the `"down"` direction.

The current status of resharding is listed in the [collection cluster info](https://api.qdrant.tech/v-1-12-x/api-reference/distributed/collection-cluster-info) which can be fetched with:

```http
GET /collections/{collection_name}/cluster
```

We always recommend to run an ongoing resharding operation till the end. But, if at any point the resharding operation needs to be aborted, you can use:

```http
POST /collections/{collection_name}/cluster
{
"abort_resharding": {}
}
```

A few things to be aware of with regards to resharding:

- during resharding, performance of your cluster may be slightly reduced
- during resharding, reported point counts will not be accurate
- resharding may be a long running operation on huge collections
- you can only run one resharding operation per collection at a time
Original file line number Diff line number Diff line change
Expand Up @@ -296,12 +296,22 @@ To ensure all nodes in your cluster are evenly utilized, the number of shards mu

> Aside: Advanced use cases such as multitenancy may require an uneven distribution of shards. See [Multitenancy](/articles/multitenancy/).

We recommend creating at least 2 shards per node to allow future expansion without having to re-shard. Re-sharding should be avoided since it requires creating a new collection. In-place re-sharding is planned for a future version of Qdrant.
We recommend creating at least 2 shards per node to allow future expansion without having to re-shard. [Resharding](#resharding) is possible when using our cloud offering, but should be avoided if hosting elsewhere as it would require creating a new collection.

If you anticipate a lot of growth, we recommend 12 shards since you can expand from 1 node up to 2, 3, 6, and 12 nodes without having to re-shard. Having more than 12 shards in a small cluster may not be worth the performance overhead.

Shards are evenly distributed across all existing nodes when a collection is first created, but Qdrant does not automatically rebalance shards if your cluster size or replication factor changes (since this is an expensive operation on large clusters). See the next section for how to move shards after scaling operations.

### Resharding

*Available as of v1.13.0 in Cloud*

Resharding allows you to change the number of shards in your existing collections if you're hosting with our [Cloud](/documentation/cloud-intro/) offering.

Resharding can change the number of shards both up and down, without having to recreate the collection from scratch.

Please refer to the [Resharding](/documentation/cloud/cluster-scaling/#resharding) section in our cloud documentation for more details.

### Moving shards

*Available as of v0.9.0*
Expand Down

0 comments on commit ead3cd8

Please sign in to comment.