Skip to content
This repository has been archived by the owner on Oct 30, 2024. It is now read-only.

feat: add progress indication for embedding generation process #127

Merged
merged 1 commit into from
Sep 19, 2024

Conversation

iwilltry42
Copy link
Collaborator

This gives some progress indication for the embeddings generation.
Unfortunately, without modifying the chromem-go code for this, we cannot get the "progress": "1/3" style output here, so maybe in Otto we can just do a counter based on the "status": "starting" with "num_documents": 3 being the total number of times it should appear.

{"time":"2024-09-19T14:56:15.374927781+02:00","level":"INFO","msg":"Adding documents to collection (generating embeddings)","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"vectorstore","vectorstore":"chromem-go","status":"starting"}
{"time":"2024-09-19T14:56:15.374966246+02:00","level":"INFO","msg":"Creating embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"starting"}
{"time":"2024-09-19T14:56:15.374975889+02:00","level":"INFO","msg":"Creating embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"starting"}
{"time":"2024-09-19T14:56:15.375093968+02:00","level":"INFO","msg":"Creating embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"starting"}
{"time":"2024-09-19T14:56:15.609623657+02:00","level":"INFO","msg":"Created embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"completed"}
{"time":"2024-09-19T14:56:15.748787889+02:00","level":"INFO","msg":"Created embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"completed"}
{"time":"2024-09-19T14:56:16.205515467+02:00","level":"INFO","msg":"Created embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"completed"}
{"time":"2024-09-19T14:56:16.205883188+02:00","level":"INFO","msg":"Added documents to collection (generated embeddings)","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"vectorstore","vectorstore":"chromem-go","status":"completed"}

@@ -160,8 +160,12 @@ func ingestPaths(ctx context.Context, c Client, opts *IngestPathsOpts, datasetID
defer sem.Release(1)

ingestedFilesCount++
currentMetadata := metadataStack[len(metadataStack)-1]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated bugfix

@iwilltry42 iwilltry42 merged commit bd17366 into main Sep 19, 2024
1 check passed
@iwilltry42 iwilltry42 deleted the feat/embedding-progress branch September 19, 2024 14:37
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants