-
Notifications
You must be signed in to change notification settings - Fork 81
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: Superduper integration (#1179)
* docs: Superduper integration * Update _index.md
- Loading branch information
Showing
2 changed files
with
88 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
87 changes: 87 additions & 0 deletions
87
qdrant-landing/content/documentation/frameworks/superduper.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,87 @@ | ||
--- | ||
title: Superduper | ||
--- | ||
|
||
# Superduper | ||
|
||
[Superduper](https://superduper.io/) is a framework for building flexible, compositional AI applications which may be applied directly to databases using a declarative programming model. These applications declare and maintain a desired state of the database, and use the database directly to store outputs of AI components, meta-data about the components and data pertaining to the state of the system. | ||
|
||
Qdrant is available as a vector search provider in Superduper. | ||
|
||
## Installation | ||
|
||
```bash | ||
pip install superduper-framework | ||
``` | ||
|
||
## Setup | ||
|
||
- To use Qdrant for vector search layer, create a `settings.yaml` file with the following config. | ||
|
||
> settings.yaml | ||
```yaml | ||
cluster: | ||
vector_search: | ||
type: qdrant | ||
|
||
vector_search_kwargs: | ||
url: "http://localhost:6333" | ||
api_key: "<YOUR_API_KEY> | ||
# Supports all parameters of qdrant_client.QdrantClient | ||
``` | ||
- Set the `SUPERDUPER_CONFIG` env value to the path of the config file. | ||
```bash | ||
export SUPERDUPER_CONFIG=path/to/settings.yaml | ||
``` | ||
That's all. You can now use Superduper backed by Qdrant. | ||
## Example | ||
Here's an example to run vector search using the configured Qdrant index. | ||
```python | ||
import json | ||
import requests | ||
from superduper import superduper, Document | ||
from superduper.ext.sentence_transformers import SentenceTransformer | ||
r = requests.get('https://superduperdb-public-demo.s3.amazonaws.com/text.json') | ||
with open('text.json', 'wb') as f: | ||
f.write(r.content) | ||
with open('text.json', 'r') as f: | ||
data = json.load(f) | ||
db = superduper('mongomock://test') | ||
_ = db['documents'].insert_many([Document({'txt': txt}) for txt in data]).execute() | ||
model = SentenceTransformer( | ||
identifier="test", | ||
predict_kwargs={"show_progress_bar": True}, | ||
model="all-MiniLM-L6-v2", | ||
device="cpu", | ||
postprocess=lambda x: x.tolist(), | ||
) | ||
|
||
vector_index = model.to_vector_index(select=db['documents'].find(), key='txt') | ||
|
||
db.apply(vector_index) | ||
|
||
query = db['documents'].like({'txt': 'Tell me about vector-search'}, vector_index=vector_index.identifier, n=3).find() | ||
cursor = query.execute() | ||
|
||
for r in cursor: | ||
print('=' * 100) | ||
print(r.unpack()['txt']) | ||
print('=' * 100) | ||
``` | ||
## 📚 Further Reading | ||
- Superduper [Intro](https://docs.superduper.io/docs/intro) | ||
- Superduper [Source](https://github.com/superduper-io/superduper) |