Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-index querying for API layer #1644

Open
wants to merge 58 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
ae8bac5
added multi-global-query function header
KennyZhang1 Jan 13, 2025
a6975f2
ported over code for merging dataframes
KennyZhang1 Jan 13, 2025
8e251af
added connection to global streaming api function
KennyZhang1 Jan 13, 2025
e33558e
Merge branch 'main' of github.com:microsoft/graphrag into dorbaker/mu…
KennyZhang1 Jan 14, 2025
96532fd
added function header for update context helper
KennyZhang1 Jan 14, 2025
2bfe76f
implemented and incorperated update_context function
KennyZhang1 Jan 14, 2025
751d67d
Updated to make sure 'parent' column in final_communities gets increm…
dorbaker Jan 14, 2025
95fee4f
first cut at multi_local_seach function
KennyZhang1 Jan 15, 2025
b0bb3fb
Merge branch 'dorbaker/multi-index-query' of github.com:microsoft/gra…
KennyZhang1 Jan 15, 2025
c7da4fa
several minor changes and fixes
KennyZhang1 Jan 15, 2025
63d2541
Updated multi index local search.
dorbaker Jan 15, 2025
8e2bf6f
Cleaned up code.
dorbaker Jan 15, 2025
428e105
fixed lambda function ruff errors
KennyZhang1 Jan 16, 2025
08c902f
fixed more ruff errors
KennyZhang1 Jan 16, 2025
af30f35
moved query api helpers to util file
KennyZhang1 Jan 16, 2025
450fea9
moved index api helpers to util file
KennyZhang1 Jan 16, 2025
a287deb
Merge branch 'main' of github.com:microsoft/graphrag into dorbaker/mu…
KennyZhang1 Jan 16, 2025
38e71ef
merged in code left out of conflict
KennyZhang1 Jan 16, 2025
5c700c3
changed GraphRagConfig object to support lists of vector stores
KennyZhang1 Jan 16, 2025
79652ed
Updated with fixes for multi_local_search.
dorbaker Jan 16, 2025
8ee8e93
Minor updates.
dorbaker Jan 17, 2025
0f792c2
Minor updates.
dorbaker Jan 17, 2025
0f949e9
Merge branch 'dorbaker/multi-index-query' of https://github.com/micro…
dorbaker Jan 17, 2025
6b790f9
Updates for ruff check.
dorbaker Jan 17, 2025
89287cb
Minor updates.
dorbaker Jan 17, 2025
282debc
removed redundant vector_store_configs arg
KennyZhang1 Jan 17, 2025
70d553f
ruff formatting changes
KennyZhang1 Jan 21, 2025
eee0f83
semversioner
KennyZhang1 Jan 21, 2025
a2e8b79
Minor fix.
dorbaker Jan 21, 2025
b4e1142
Merge branch 'dorbaker/multi-index-query' of github.com:microsoft/gra…
KennyZhang1 Jan 21, 2025
6b05450
spellcheck fixes
KennyZhang1 Jan 21, 2025
4c60dd1
ruff
KennyZhang1 Jan 21, 2025
8ce13d9
test fix for cicd errors
KennyZhang1 Jan 21, 2025
3052f8c
another test fix
KennyZhang1 Jan 21, 2025
77de9aa
added explicit typing for ci tests
KennyZhang1 Jan 21, 2025
2b88b8d
added dict type check for vector_store during indexing
KennyZhang1 Jan 21, 2025
3b93d0d
more ruff fixes
KennyZhang1 Jan 21, 2025
b157c42
moved type check
KennyZhang1 Jan 21, 2025
e307956
Removed streaming. Added multi drift and basic searches.
dorbaker Jan 21, 2025
b69196b
Merge branch 'dorbaker/multi-index-query' of https://github.com/micro…
dorbaker Jan 21, 2025
7c4310f
Formatting changes.
dorbaker Jan 21, 2025
e6ba9e3
Updates for pyright.
dorbaker Jan 21, 2025
818b31e
Update for ruff.
dorbaker Jan 21, 2025
1a3014a
Ruff formatted.
dorbaker Jan 21, 2025
7702271
Merged in main and attempt to make work.
dorbaker Jan 22, 2025
11fd956
first cut at fixing vector store typing errors
KennyZhang1 Jan 22, 2025
648c9d3
got multi local search working with new config
KennyZhang1 Jan 22, 2025
2d13429
ruff and test fixes
KennyZhang1 Jan 22, 2025
2d35af1
added fix for embeddings type error
KennyZhang1 Jan 22, 2025
5efeb74
renamed multi index api functions
KennyZhang1 Jan 24, 2025
e1cc16f
ruff
KennyZhang1 Jan 24, 2025
9b35e09
convert config model to dict[VectorStoreConfig]
KennyZhang1 Jan 24, 2025
7011316
Merge branch 'main' of github.com:microsoft/graphrag into dorbaker/mu…
KennyZhang1 Jan 24, 2025
60fcbe7
modified tests to support new vector_store model
KennyZhang1 Jan 24, 2025
a260c21
ruff fixes
KennyZhang1 Jan 24, 2025
a4a9c21
changed some test setups to match new model
KennyZhang1 Jan 24, 2025
d44b459
changed ci/cd settings files to match new structure
KennyZhang1 Jan 24, 2025
c6d491c
Fix stderror check
AlonsoGuevara Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .semversioner/next-release/patch-20250121150223319652.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"type": "patch",
"description": "implemented multi-index querying for api layer"
}
20 changes: 2 additions & 18 deletions graphrag/api/index.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
from graphrag.index.run.run_workflows import run_workflows
from graphrag.index.typing import PipelineRunResult
from graphrag.logger.base import ProgressLogger
from graphrag.utils.api import get_workflows_list

log = logging.getLogger(__name__)

Expand Down Expand Up @@ -60,7 +61,7 @@ async def build_index(
if memory_profile:
log.warning("New pipeline does not yet support memory profiling.")

workflows = _get_workflows_list(config)
workflows = get_workflows_list(config)

async for output in run_workflows(
workflows,
Expand All @@ -79,20 +80,3 @@ async def build_index(
progress_logger.info(str(output.result))

return outputs


def _get_workflows_list(config: GraphRagConfig) -> list[str]:
return [
"create_base_text_units",
"create_final_documents",
"extract_graph",
"compute_communities",
"create_final_entities",
"create_final_relationships",
"create_final_nodes",
"create_final_communities",
*(["create_final_covariates"] if config.claim_extraction.enabled else []),
"create_final_text_units",
"create_final_community_reports",
"generate_text_embeddings",
]
Loading
Loading