-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Initial refactor * Add arguments aliases and a debugger option * Fix merge issue * remove extra parameters * refactor * Move changes to a new project * Setup test reporter * Fix performance tests * Remove unnecessary stuff * Set up global benchmark script * Update readme * Adjust pipeline * Fix output path * Remove test results file * Fix formatting issue * Fix eslint config * Save main benchmark results only when run on master branch * Fix condition * Fix eslint problems * Remove start script * Change file structure * Use public query executor * Refactor blocking stats * Add table dependency * Remove logging from hierarchy provider * Improve test reporter console output * Remove check leaks option * Remove pid from profile name * Remove pre-caching of iModels * Simplify float rounding * Fix p95 * Add additional entries for benchmark * Fix import being on top of header * Fix debugger crashing on startup * Fix issue of TestReporter measuring time from `beforeEach` to test end. * Add undefined check * Improve comment * Rename itMeasures to run * Improve text for blocking benchmark entries --------- Co-authored-by: Dmitrij Kuzmiciov <[email protected]>
- Loading branch information
Showing
20 changed files
with
1,215 additions
and
799 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
datasets/* | ||
benchmark.json | ||
test-results.xml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
{ | ||
"timeout": 60000, | ||
"file": "./lib/main.js", | ||
"reporter": ["./lib/util/TestReporter.js"] | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
# Presentation performance tests | ||
|
||
## Tests | ||
|
||
The tests are supposed to represent various scenarios that we want to profile. | ||
|
||
## Test reporter | ||
|
||
Additionally, we want to measure how much time the main thread is being blocked. | ||
Also, these tests have a different purpose - to provide a benchmark that will be used by GitHub actions and can be useful for the developers. | ||
The simplest way to accommodate that is to use a custom test reporter (defined in `TestReporter.ts`). | ||
The reporter gathers test durations and information about main thread blocking and saves it to a file if an output path is provided. | ||
|
||
Example: `mocha -R ./lib/TestReporter.js -O BENCHMARK_OUTPUT_PATH="./results.json"` | ||
|
||
### iModels | ||
|
||
The tests may use iModels that are managed in `Datasets.ts` module. The iModels are stored locally in the `./datasets` folder. | ||
|
||
## Usage | ||
|
||
- In order to run all performance tests type: | ||
`pnpm test` | ||
- In order to run performance tests and save the results to `./benchmark.json` enter: | ||
`pnpm benchmark`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
/*--------------------------------------------------------------------------------------------- | ||
* Copyright (c) Bentley Systems, Incorporated. All rights reserved. | ||
* See LICENSE.md in the project root for license terms and full copyright notice. | ||
*--------------------------------------------------------------------------------------------*/ | ||
const iTwinPlugin = require("@itwin/eslint-plugin"); | ||
const eslintBaseConfig = require("../../eslint.base.config"); | ||
|
||
module.exports = [ | ||
{ | ||
files: ["**/*.ts"], | ||
...iTwinPlugin.configs.iTwinjsRecommendedConfig, | ||
}, | ||
...eslintBaseConfig, | ||
{ | ||
files: ["**/*.ts"], | ||
rules: { | ||
"no-console": "off", | ||
}, | ||
}, | ||
]; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
{ | ||
"name": "presentation-performance-tests", | ||
"version": "0.0.0", | ||
"private": true, | ||
"scripts": { | ||
"test": "NODE_OPTIONS=\"--enable-source-maps\" mocha --config ./.mocharc.json ./lib/*test.js", | ||
"benchmark": "npm run test -- -O BENCHMARK_OUTPUT_PATH=./benchmark.json", | ||
"build": "tsc", | ||
"clean": "rimraf lib temp", | ||
"lint": "eslint \"./src/**/*.ts\"" | ||
}, | ||
"dependencies": { | ||
"@itwin/core-backend": "^4.4.0", | ||
"@itwin/core-bentley": "^4.4.0", | ||
"@itwin/core-common": "^4.4.0", | ||
"@itwin/ecschema-metadata": "^4.4.0", | ||
"@itwin/presentation-hierarchy-builder": "workspace:*", | ||
"@itwin/presentation-core-interop": "workspace:*", | ||
"@itwin/presentation-models-tree": "workspace:*", | ||
"as-table": "^1.0.55", | ||
"mocha": "^10.3.0", | ||
"blocked": "^1.3.0", | ||
"rxjs": "^7.8.1" | ||
}, | ||
"devDependencies": { | ||
"@itwin/eslint-plugin": "4.0.0-dev.48", | ||
"@types/blocked": "^1.3.4", | ||
"@types/mocha": "^10.0.6", | ||
"@types/node": "^18.17.7", | ||
"eslint": "^8.56.0", | ||
"rimraf": "^5.0.5", | ||
"typescript": "~5.0.4" | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
/*--------------------------------------------------------------------------------------------- | ||
* Copyright (c) Bentley Systems, Incorporated. All rights reserved. | ||
* See LICENSE.md in the project root for license terms and full copyright notice. | ||
*--------------------------------------------------------------------------------------------*/ | ||
import fs from "fs"; | ||
import path from "path"; | ||
|
||
async function downloadDataset(name: string, downloadUrl: string, localPath: string): Promise<void> { | ||
console.log(`Downloading "${name}" iModel from "${downloadUrl}"...`); | ||
const response = await fetch(downloadUrl); | ||
if (!response.ok) { | ||
throw new Error(`Failed to fetch ${name} iModel: ${response.statusText}`); | ||
} | ||
|
||
await response.body!.pipeTo(fs.WriteStream.toWeb(fs.createWriteStream(localPath))); | ||
} | ||
|
||
/** Paths to downloaded iModels. */ | ||
export const iModelPaths = new Array<string>(); | ||
|
||
/** Loads iModels into cache for the tests to use. */ | ||
export async function loadDataSets(datasetsDirPath: string) { | ||
await fs.promises.mkdir(datasetsDirPath, { recursive: true }); | ||
|
||
const datasets = [["Baytown", "https://github.com/imodeljs/desktop-starter/raw/master/assets/Baytown.bim"]].map((entry) => [ | ||
...entry, | ||
path.join(datasetsDirPath, `${entry[0]}.bim`), | ||
]); | ||
|
||
const datasetPaths = await Promise.all( | ||
datasets.map(async ([name, url, localPath]) => { | ||
try { | ||
await fs.promises.access(localPath, fs.constants.F_OK); | ||
} catch { | ||
await downloadDataset(name, url, localPath); | ||
} | ||
return path.resolve(localPath); | ||
}), | ||
); | ||
|
||
iModelPaths.push(...datasetPaths); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,84 @@ | ||
/*--------------------------------------------------------------------------------------------- | ||
* Copyright (c) Bentley Systems, Incorporated. All rights reserved. | ||
* See LICENSE.md in the project root for license terms and full copyright notice. | ||
*--------------------------------------------------------------------------------------------*/ | ||
import { expand, filter, from, mergeAll, of } from "rxjs"; | ||
import { IModelDb } from "@itwin/core-backend"; | ||
import { ISchemaLocater, Schema, SchemaContext, SchemaInfo, SchemaKey, SchemaMatchType } from "@itwin/ecschema-metadata"; | ||
import { createECSqlQueryExecutor, createMetadataProvider } from "@itwin/presentation-core-interop"; | ||
import { createLimitingECSqlQueryExecutor, HierarchyNode, HierarchyProvider } from "@itwin/presentation-hierarchy-builder"; | ||
import { ModelsTreeDefinition } from "@itwin/presentation-models-tree"; | ||
|
||
export class StatelessHierarchyProvider { | ||
private readonly _provider: HierarchyProvider; | ||
|
||
constructor( | ||
iModelDb: IModelDb, | ||
private readonly _nodeRequestLimit = 10, | ||
) { | ||
this._provider = createProvider(iModelDb); | ||
} | ||
|
||
public async loadInitialHierarchy(): Promise<void> { | ||
await this.loadNodes((node) => node.children && !!node.autoExpand); | ||
} | ||
|
||
public async loadFullHierarchy(): Promise<void> { | ||
await this.loadNodes((node) => node.children); | ||
} | ||
|
||
private async loadNodes(nodeHasChildren: (node: HierarchyNode) => boolean) { | ||
await new Promise<void>((resolve, reject) => { | ||
const nodesObservable = of<HierarchyNode | undefined>(undefined).pipe( | ||
expand((parentNode) => { | ||
return from(this._provider.getNodes({ parentNode })).pipe( | ||
mergeAll(), | ||
filter((node) => nodeHasChildren(node)), | ||
); | ||
}, this._nodeRequestLimit), | ||
); | ||
nodesObservable.subscribe({ | ||
complete: resolve, | ||
error: reject, | ||
}); | ||
}); | ||
} | ||
} | ||
|
||
function createProvider(iModelDb: IModelDb) { | ||
const schemas = new SchemaContext(); | ||
const locater = new SchedulingSchemaLocater(iModelDb); | ||
schemas.addLocater(locater); | ||
const metadataProvider = createMetadataProvider(schemas); | ||
|
||
return new HierarchyProvider({ | ||
metadataProvider, | ||
hierarchyDefinition: new ModelsTreeDefinition({ metadataProvider }), | ||
queryExecutor: createLimitingECSqlQueryExecutor(createECSqlQueryExecutor(iModelDb), 1000), | ||
}); | ||
} | ||
|
||
class SchedulingSchemaLocater implements ISchemaLocater { | ||
constructor(private readonly _iModelDb: IModelDb) {} | ||
|
||
public getSchemaSync<T extends Schema>(_schemaKey: Readonly<SchemaKey>, _matchType: SchemaMatchType, _schemaContext: SchemaContext): T | undefined { | ||
console.error(`getSchemaSync not implemented`); | ||
return undefined; | ||
} | ||
|
||
public async getSchemaInfo(schemaKey: Readonly<SchemaKey>, matchType: SchemaMatchType, schemaContext: SchemaContext): Promise<SchemaInfo | undefined> { | ||
const schemaJson = this._iModelDb.getSchemaProps(schemaKey.name); | ||
const schemaInfo = await Schema.startLoadingFromJson(schemaJson, schemaContext); | ||
if (schemaInfo !== undefined && schemaInfo.schemaKey.matches(schemaKey, matchType)) { | ||
return schemaInfo; | ||
} | ||
return undefined; | ||
} | ||
|
||
public async getSchema<T extends Schema>(schemaKey: Readonly<SchemaKey>, matchType: SchemaMatchType, schemaContext: SchemaContext): Promise<T | undefined> { | ||
await this.getSchemaInfo(schemaKey, matchType, schemaContext); | ||
// eslint-disable-next-line @itwin/no-internal | ||
const schema = await schemaContext.getCachedSchema(schemaKey, matchType); | ||
return schema as T; | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
/*--------------------------------------------------------------------------------------------- | ||
* Copyright (c) Bentley Systems, Incorporated. All rights reserved. | ||
* See LICENSE.md in the project root for license terms and full copyright notice. | ||
*--------------------------------------------------------------------------------------------*/ | ||
import { IModelHost } from "@itwin/core-backend"; | ||
import { loadDataSets } from "./Datasets"; | ||
|
||
before(async () => { | ||
await IModelHost.startup({ | ||
profileName: "presentation-performance-tests", | ||
}); | ||
await loadDataSets("./datasets"); | ||
}); | ||
|
||
after(async () => { | ||
await IModelHost.shutdown(); | ||
}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
/*--------------------------------------------------------------------------------------------- | ||
* Copyright (c) Bentley Systems, Incorporated. All rights reserved. | ||
* See LICENSE.md in the project root for license terms and full copyright notice. | ||
*--------------------------------------------------------------------------------------------*/ | ||
import { SnapshotDb } from "@itwin/core-backend"; | ||
import { iModelPaths } from "./Datasets"; | ||
import { StatelessHierarchyProvider } from "./StatelessHierarchyProvider"; | ||
import { run } from "./util/TestUtilities"; | ||
|
||
describe("stateless hierarchy", () => { | ||
let iModel: SnapshotDb; | ||
|
||
beforeEach(() => { | ||
iModel = SnapshotDb.openFile(iModelPaths[0]); | ||
}); | ||
|
||
afterEach(() => { | ||
iModel.close(); | ||
}); | ||
|
||
run("loads initial hierarchy", async () => { | ||
const provider = new StatelessHierarchyProvider(iModel); | ||
await provider.loadInitialHierarchy(); | ||
}); | ||
|
||
run("loads full hierarchy", async () => { | ||
const provider = new StatelessHierarchyProvider(iModel); | ||
await provider.loadFullHierarchy(); | ||
}); | ||
}); |
Oops, something went wrong.