Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exp: Reduce pool size in prune #595

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft

Exp: Reduce pool size in prune #595

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
95b9177
Exp: Reduce pool size in prune
Oct 29, 2024
85a0826
post build cleanup untouched
Oct 30, 2024
94b0803
add flag for experiments
Nov 4, 2024
8f5ac12
remove flags
Nov 5, 2024
baeb526
fix formatting
Nov 5, 2024
af32dd3
update alpha setup
Nov 5, 2024
b2abb2c
add prec40M results
Nov 5, 2024
86640b0
add prec40M build and search results
Nov 5, 2024
886952f
add prec40M baseline results
Nov 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,9 @@ FakesAssemblies/
.ntvs_analysis.dat
node_modules/

# Visual Studio Settings
.vscode

# Visual Studio 6 build log
*.plg

Expand Down
1 change: 1 addition & 0 deletions apps/build_memory_index.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ int main(int argc, char **argv)
program_options_utils::FILTERED_LBUILD);
optional_configs.add_options()("label_type", po::value<std::string>(&label_type)->default_value("uint"),
program_options_utils::LABEL_TYPE_DESCRIPTION);
//optional_configs.add_options()("reduce_pool", po::bool_switch(&reduce_pool)->default_value(false),"Reduce Pool");

// Merge required and optional parameters
desc.add(required_configs).add(optional_configs);
Expand Down
7 changes: 4 additions & 3 deletions include/index.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
#define EXPAND_IF_FULL 0
#define DEFAULT_MAXC 750

//extern bool reduce_pool;
namespace diskann
{

Expand Down Expand Up @@ -265,17 +266,17 @@ template <typename T, typename TagT = uint32_t, typename LabelT = uint32_t> clas
uint32_t filteredLindex = 0);

void prune_neighbors(const uint32_t location, std::vector<Neighbor> &pool, std::vector<uint32_t> &pruned_list,
InMemQueryScratch<T> *scratch);
InMemQueryScratch<T> *scratch, bool reduce_pool);

void prune_neighbors(const uint32_t location, std::vector<Neighbor> &pool, const uint32_t range,
const uint32_t max_candidate_size, const float alpha, std::vector<uint32_t> &pruned_list,
InMemQueryScratch<T> *scratch);
InMemQueryScratch<T> *scratch, bool reduce_pool);

// Prunes candidates in @pool to a shorter list @result
// @pool must be sorted before calling
void occlude_list(const uint32_t location, std::vector<Neighbor> &pool, const float alpha, const uint32_t degree,
const uint32_t maxc, std::vector<uint32_t> &result, InMemQueryScratch<T> *scratch,
const tsl::robin_set<uint32_t> *const delete_set_ptr = nullptr);
const tsl::robin_set<uint32_t> *const delete_set_ptr = nullptr, bool reduce_pool = false);

// add reverse links from all the visited nodes to node n.
void inter_insert(uint32_t n, std::vector<uint32_t> &pruned_list, const uint32_t range,
Expand Down
84 changes: 84 additions & 0 deletions prec40M/baseline_r64_k10.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
Reading (with alignment) bin file /nvmessd1/fbv4/queries384d.bin ...Metadata: #pts = 13265, #dims = 384, aligned_dim = 384... allocating aligned memory of 5093760 bytes... done. Copying data to mem_aligned buffer... done.
Opened: /nvmessd1/fbv4/gt100_prec40M, size: 10612008, cache_size: 10612008
Reading truthset file /nvmessd1/fbv4/gt100_prec40M ...
Metadata: #pts = 13265, #dims = 100...
Using AVX2 distance computation DistanceL2Int8.
Resizing took: 1.40021s
From graph header, expected_file_size: 7438757948, _max_observed_degree: 64, _start: 11890341, file_frozen_pts: 0
Loading vamana graph /nvmessd1/fbv4/avarhade/prec40M_memory_index_r64_l100.......done. Index has 40469609 nodes and 1819219872 out-edges, _start is set to 11890341
Num frozen points:0 _nd: 40469609 _start: 11890341 size(_location_to_tag): 0 size(_tag_to_location):0 Max points: 40469609
Index loaded
Using 48 threads to search
Ls QPS Avg dist cmps Mean Latency (mus) 99.9 Latency Recall@10
=================================================================================
10 51557.93 1324.67 913.18 9792.77 43.13
20 33640.81 2021.14 1420.26 3249.95 58.38
30 25210.59 2617.97 1896.59 4469.27 66.24
40 20547.19 3168.06 2326.82 5450.12 71.25
50 17328.33 3687.22 2760.19 6599.82 74.44
60 14980.50 4195.96 3193.32 7739.67 76.98
70 13341.03 4685.40 3588.47 8306.07 78.78
80 11462.47 5171.15 4176.58 11172.33 80.21
90 10723.69 5654.48 4462.36 10865.74 81.53
100 9782.57 6129.37 4892.21 12162.22 82.58
Done searching. Now saving results
Writing bin: /home/rakri/avarhade/Dump/tmp_10_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_10_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_20_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_20_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_30_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_30_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_40_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_40_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_50_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_50_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_60_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_60_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_70_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_70_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_80_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_80_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_90_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_90_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_100_idx_uint32.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_100_dists_float.bin
bin: #pts = 13265, #dims = 10, size = 530608B
Finished writing bin.
21 changes: 21 additions & 0 deletions prec40M/baseline_r64_k100.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
Reading (with alignment) bin file /nvmessd1/fbv4/queries384d.bin ...Metadata: #pts = 13265, #dims = 384, aligned_dim = 384... allocating aligned memory of 5093760 bytes... done. Copying data to mem_aligned buffer... done.
Opened: /nvmessd1/fbv4/gt100_prec40M, size: 10612008, cache_size: 10612008
Reading truthset file /nvmessd1/fbv4/gt100_prec40M ...
Metadata: #pts = 13265, #dims = 100...
Using AVX2 distance computation DistanceL2Int8.
Resizing took: 1.40809s
From graph header, expected_file_size: 7438757948, _max_observed_degree: 64, _start: 11890341, file_frozen_pts: 0
Loading vamana graph /nvmessd1/fbv4/avarhade/prec40M_memory_index_r64_l100.......done. Index has 40469609 nodes and 1819219872 out-edges, _start is set to 11890341
Num frozen points:0 _nd: 40469609 _start: 11890341 size(_location_to_tag): 0 size(_tag_to_location):0 Max points: 40469609
Index loaded
Using 48 threads to search
Ls QPS Avg dist cmps Mean Latency (mus) 99.9 Latency Recall@100
=================================================================================
100 10331.70 6129.37 4626.05 11214.14 66.81
Done searching. Now saving results
Writing bin: /home/rakri/avarhade/Dump/tmp_100_idx_uint32.bin
bin: #pts = 13265, #dims = 100, size = 5306008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_100_dists_float.bin
bin: #pts = 13265, #dims = 100, size = 5306008B
Finished writing bin.
70 changes: 70 additions & 0 deletions prec40M/baseline_r64_k25.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
Reading (with alignment) bin file /nvmessd1/fbv4/queries384d.bin ...Metadata: #pts = 13265, #dims = 384, aligned_dim = 384... allocating aligned memory of 5093760 bytes... done. Copying data to mem_aligned buffer... done.
Opened: /nvmessd1/fbv4/gt100_prec40M, size: 10612008, cache_size: 10612008
Reading truthset file /nvmessd1/fbv4/gt100_prec40M ...
Metadata: #pts = 13265, #dims = 100...
Using AVX2 distance computation DistanceL2Int8.
Resizing took: 1.48129s
From graph header, expected_file_size: 7438757948, _max_observed_degree: 64, _start: 11890341, file_frozen_pts: 0
Loading vamana graph /nvmessd1/fbv4/avarhade/prec40M_memory_index_r64_l100.......done. Index has 40469609 nodes and 1819219872 out-edges, _start is set to 11890341
Num frozen points:0 _nd: 40469609 _start: 11890341 size(_location_to_tag): 0 size(_tag_to_location):0 Max points: 40469609
Index loaded
Using 48 threads to search
Ls QPS Avg dist cmps Mean Latency (mus) 99.9 Latency Recall@25
=================================================================================
30 25253.53 2617.97 1885.01 10497.99 60.18
40 20654.60 3168.06 2316.21 5201.40 65.38
50 17413.17 3687.22 2747.47 6720.12 68.82
60 15010.45 4195.96 3187.87 7331.08 71.60
70 13300.10 4685.40 3599.56 8146.48 73.66
80 11990.02 5171.15 3991.75 8909.45 75.35
90 10918.86 5654.48 4385.05 9846.92 76.87
100 9976.54 6129.37 4799.29 10763.89 78.14
Done searching. Now saving results
Writing bin: /home/rakri/avarhade/Dump/tmp_30_idx_uint32.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_30_dists_float.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_40_idx_uint32.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_40_dists_float.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_50_idx_uint32.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_50_dists_float.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_60_idx_uint32.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_60_dists_float.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_70_idx_uint32.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_70_dists_float.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_80_idx_uint32.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_80_dists_float.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_90_idx_uint32.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_90_dists_float.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_100_idx_uint32.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_100_dists_float.bin
bin: #pts = 13265, #dims = 25, size = 1326508B
Finished writing bin.
56 changes: 56 additions & 0 deletions prec40M/baseline_r64_k50.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
Reading (with alignment) bin file /nvmessd1/fbv4/queries384d.bin ...Metadata: #pts = 13265, #dims = 384, aligned_dim = 384... allocating aligned memory of 5093760 bytes... done. Copying data to mem_aligned buffer... done.
Opened: /nvmessd1/fbv4/gt100_prec40M, size: 10612008, cache_size: 10612008
Reading truthset file /nvmessd1/fbv4/gt100_prec40M ...
Metadata: #pts = 13265, #dims = 100...
Using AVX2 distance computation DistanceL2Int8.
Resizing took: 1.41607s
From graph header, expected_file_size: 7438757948, _max_observed_degree: 64, _start: 11890341, file_frozen_pts: 0
Loading vamana graph /nvmessd1/fbv4/avarhade/prec40M_memory_index_r64_l100.......done. Index has 40469609 nodes and 1819219872 out-edges, _start is set to 11890341
Num frozen points:0 _nd: 40469609 _start: 11890341 size(_location_to_tag): 0 size(_tag_to_location):0 Max points: 40469609
Index loaded
Using 48 threads to search
Ls QPS Avg dist cmps Mean Latency (mus) 99.9 Latency Recall@50
=================================================================================
50 17629.05 3687.22 2705.62 14582.74 62.96
60 14420.19 4195.96 3319.41 9059.50 65.92
70 12430.30 4685.40 3850.11 10474.53 68.19
80 12256.81 5171.15 3903.45 9068.07 70.07
90 11310.12 5654.48 4230.41 9502.88 71.76
100 10462.90 6129.37 4574.90 9994.75 73.17
Done searching. Now saving results
Writing bin: /home/rakri/avarhade/Dump/tmp_50_idx_uint32.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_50_dists_float.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_60_idx_uint32.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_60_dists_float.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_70_idx_uint32.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_70_dists_float.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_80_idx_uint32.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_80_dists_float.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_90_idx_uint32.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_90_dists_float.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_100_idx_uint32.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Writing bin: /home/rakri/avarhade/Dump/tmp_100_dists_float.bin
bin: #pts = 13265, #dims = 50, size = 2653008B
Finished writing bin.
Loading
Loading