Skip to content

Commit

Permalink
Wrong dependency resolution plus performance optimizations when build…
Browse files Browse the repository at this point in the history 
…ing (#3757)

Two new parameter in Sort-AppFoldersByDependencies -
onlyTheseAppFoldersPlusDepending and SkipApps. If specified, then only
the appFolders specified in the first array plus the apps depending on
these recursively will be sorted. The remaining folder not included in
the sorted appfolders list will be returned in skipApps. This is used
for incrementals builds in AL-Go for GitHub.

Remove some excessive logging

If Import-TestToolkitToBcContainer was running with useCompilerFolder,
then it would require a container to be present at that time.

Make Run-AlPipeline do just-in-time creation of CompilerFolder and
Containers - meaning that if you are using useCompilerFolder and do not
have doNotPublishApps set - it will create a compilerFolder and compile
all apps using that, and afterwards - it will create the container,
publish the apps and run the tests.

Also...
Fixes microsoft/AL-Go#1338

---------

Co-authored-by: freddydk <[email protected]>
  • Loading branch information
@freddydk
freddydk and freddydk authored Dec 12, 2024
1 parent f4791a3 commit 7c8e804
Show file tree
Hide file tree
Showing 8 changed files with 619 additions and 416 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/CI.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ jobs:
}
Linux:
runs-on: [ ubuntu-latest ]
runs-on: [ ubuntu-24.04 ]
defaults:
run:
shell: pwsh
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/RunTests.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ jobs:
}
Linux:
runs-on: [ ubuntu-latest ]
runs-on: [ ubuntu-24.04 ]
needs: [ AnalyzeTests ]
if: needs.AnalyzeTests.outputs.linuxtests != '[]'
strategy:
Expand Down
940 changes: 546 additions & 394 deletions AppHandling/Run-AlPipeline.ps1
View file

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions AppHandling/Sort-AppFilesByDependencies.ps1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,8 @@ function Sort-AppFilesByDependencies {
}
else {
if (-not ($script:unresolvedDependencies | Where-Object { $_ } | Where-Object { "$(if ($_.PSObject.Properties.name -eq 'AppId') { $_.AppId } else { $_.Id })" -eq $dependencyAppId })) {
$appFileName = "$($dependency.publisher)_$($dependency.name)_$($dependency.version)).app".Split([System.IO.Path]::GetInvalidFileNameChars()) -join ''
if ($dependencyAppid -ne '63ca2fa4-4f03-4f2b-a480-172fef340d3f' -and $dependencyAppId -ne '437dbf0e-84ff-417a-965d-ed2bb9650972') {
$appFileName = "$($dependency.publisher)_$($dependency.name)_$($dependency.version).app".Split([System.IO.Path]::GetInvalidFileNameChars()) -join ''
if ($dependencyAppid -ne '63ca2fa4-4f03-4f2b-a480-172fef340d3f' -and $dependencyAppId -ne '437dbf0e-84ff-417a-965d-ed2bb9650972' -and $dependencyAppId -ne 'f3552374-a1f2-4356-848e-196002525837') {
Write-Warning "Dependency $($dependencyAppId):$appFileName not found"
}
$script:unresolvedDependencies += @($dependency)
Expand Down
54 changes: 45 additions & 9 deletions AppHandling/Sort-AppFoldersByDependencies.ps1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@
If specified, this reference parameter will contain unresolved dependencies after sorting
.Parameter knownApps
If specified, this reference parameter will contain all known appids
.Parameter skippedApps
If specified, this reference parameter will contain all skipped appids
.Parameter selectSubordinates
If specified, this is the list of appFolders to include (together with subordinates - i.e. appFolders depending on these appFolders)
.Example
$folders = Sort-AppFoldersByDependencies -appFolders @($folder1, $folder2)
#>
Expand All @@ -23,7 +27,10 @@ function Sort-AppFoldersByDependencies {
[Parameter(Mandatory=$false)]
[ref] $unknownDependencies,
[Parameter(Mandatory=$false)]
[ref] $knownApps
[ref] $knownApps,
[Parameter(Mandatory=$false)]
[ref] $skippedApps,
[string[]] $selectSubordinates = @()
)

$telemetryScope = InitTelemetryScope -name $MyInvocation.InvocationName -parameterValues $PSBoundParameters -includeParameters @()
Expand All @@ -40,6 +47,7 @@ try {
# Read all app.json objects, populate $apps
$apps = $()
$folders = @{}
$script:includeAppIds = @()
$appFolders | ForEach-Object {
$appFolder = "$baseFolder$_"
$appJsonFile = Join-Path $appFolder "app.json"
Expand Down Expand Up @@ -68,14 +76,20 @@ try {
$appJson | Add-Member -Name "dependencies" -Type NoteProperty -Value @()
}
if ($appJson.psobject.Members | Where-Object name -eq "application") {
if ($appJson.Id -ne "63ca2fa4-4f03-4f2b-a480-172fef340d3f") {
if ($appJson.Id -ne "63ca2fa4-4f03-4f2b-a480-172fef340d3f" -and $appJson.Id -ne "f3552374-a1f2-4356-848e-196002525837" -and $appJson.Id -ne "437dbf0e-84ff-417a-965d-ed2bb9650972") {
$appJson.dependencies += @( New-Object psobject -Property ([ordered]@{ "appId" = "437dbf0e-84ff-417a-965d-ed2bb9650972"; "publisher" = "Microsoft"; "name" = "Base Application"; "version" = $appJson.application }) )
$appJson.dependencies += @( New-Object psobject -Property ([ordered]@{ "appId" = "63ca2fa4-4f03-4f2b-a480-172fef340d3f"; "publisher" = "Microsoft"; "name" = "System Application"; "version" = $appJson.application }) )
if ([System.Version]$appJson.application -ge [System.Version]"24.0.0.0") {
$appJson.dependencies += @( New-Object psobject -Property ([ordered]@{ "appId" = "f3552374-a1f2-4356-848e-196002525837"; "publisher" = "Microsoft"; "name" = "Business Foundation"; "version" = $appJson.application }) )
}
}
}

$folders += @{ "$($appJson.Id):$($appJson.Version)" = $appFolder }
$apps += @($appJson)
if ($selectSubordinates -contains $_) {
$script:includeAppIds += @($appJson.Id)
}
}
}

Expand All @@ -84,50 +98,72 @@ try {
$script:unresolvedDependencies = $()

function AddAnApp { Param($anApp)
$includeThis = $false
$alreadyAdded = $script:sortedApps | Where-Object { $_.Id -eq $anApp.Id }
if (-not ($alreadyAdded)) {
AddDependencies -anApp $anApp
if (AddDependencies -anApp $anApp) {
if ($script:includeAppIds -notcontains $anApp.Id) {
$script:includeAppIds += @($anApp.Id)
}
$includeThis = $true
}
$script:sortedApps += $anApp
}
return $includeThis
}

function AddDependency { Param($dependency)
$dependencyAppId = "$(if ($dependency.PSObject.Properties.name -eq 'AppId') { $dependency.AppId } else { $dependency.Id })"
$includeThis = $script:includeAppIds -contains $dependencyAppId
$dependentApp = $apps | Where-Object { $_.Id -eq $dependencyAppId } | Sort-Object -Property @{ "Expression" = "[System.Version]Version" }
if ($dependentApp) {
if ($dependentApp -is [Array]) {
Write-Host -ForegroundColor Yellow "AppFiles contains multiple versions of the app with AppId $dependencyAppId"
$dependentApp = $dependentApp | Select-Object -Last 1
}
AddAnApp -AnApp $dependentApp
if (AddAnApp -AnApp $dependentApp) {
$includeThis = $true
}
}
else {
if (-not ($script:unresolvedDependencies | Where-Object { $_ } | Where-Object { "$(if ($_.PSObject.Properties.name -eq 'AppId') { $_.AppId } else { $_.Id })" -eq $dependencyAppId })) {
$appFileName = "$($dependency.publisher)_$($dependency.name)_$($dependency.version)).app".Split([System.IO.Path]::GetInvalidFileNameChars()) -join ''
if ($dependencyAppid -ne '63ca2fa4-4f03-4f2b-a480-172fef340d3f' -and $dependencyAppId -ne '437dbf0e-84ff-417a-965d-ed2bb9650972') {
$appFileName = "$($dependency.publisher)_$($dependency.name)_$($dependency.version).app".Split([System.IO.Path]::GetInvalidFileNameChars()) -join ''
if ($dependencyAppid -ne '63ca2fa4-4f03-4f2b-a480-172fef340d3f' -and $dependencyAppId -ne '437dbf0e-84ff-417a-965d-ed2bb9650972' -and $dependencyAppId -ne 'f3552374-a1f2-4356-848e-196002525837') {
Write-Warning "Dependency $($dependencyAppId):$appFileName not found"
}
$script:unresolvedDependencies += @($dependency)
}
}
return $includeThis
}

function AddDependencies { Param($anApp)
$includeThis = $false
if ($anApp) {
if ($anApp.psobject.Members | Where-Object name -eq "dependencies") {
if ($anApp.Dependencies) {
$anApp.Dependencies | ForEach-Object { AddDependency -Dependency $_ }
$anApp.Dependencies | ForEach-Object {
if (AddDependency -Dependency $_) {
$includeThis = $true
}
}
}
}
}
return $includeThis
}

$apps | Where-Object { $_.Name -eq "Application" } | ForEach-Object { AddAnApp -anApp $_ }
$apps | ForEach-Object { AddAnApp -AnApp $_ }
$apps | Where-Object { $_.Name -eq "Application" } | ForEach-Object { AddAnApp -anApp $_ | Out-Null }
$apps | ForEach-Object { AddAnApp -AnApp $_ | Out-Null }

$script:sortedApps | ForEach-Object {
($folders["$($_.id):$($_.version)"]).SubString($baseFolder.Length)
}
if ($skippedApps -and $selectSubordinates) {
$skippedApps.value = $script:sortedApps | Where-Object { $script:includeAppIds -notcontains $_.id } | ForEach-Object {
($folders["$($_.id):$($_.version)"]).SubString($baseFolder.Length)
}
}
if ($knownApps) {
$knownApps.value += @($script:sortedApps | ForEach-Object {
$_.Id
Expand Down
16 changes: 9 additions & 7 deletions HelperFunctions.ps1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1055,7 +1055,7 @@ function GetAppInfo {
}
Set-Location (Split-Path $cacheAppInfoPath -parent)
}
Write-GroupStart -Message "Getting .app info $cacheAppInfoPath"
Write-Host "Getting .app info $cacheAppInfoPath"
$binPath = Join-Path $compilerFolder 'compiler/extension/bin'
$alToolDll = ''
if ($isLinux) {
Expand Down Expand Up @@ -1107,11 +1107,9 @@ function GetAppInfo {
$package = $null
try {
foreach($path in $appFiles) {
Write-Host -NoNewline "- $([System.IO.Path]::GetFileName($path))"
$relativePath = Resolve-Path -Path $path -Relative
if ($appInfoCache -and $appInfoCache.PSObject.Properties.Name -eq $relativePath) {
$appInfo = $appInfoCache."$relativePath"
Write-Host " (cached)"
}
else {
if ($alToolExists) {
Expand All @@ -1130,7 +1128,6 @@ function GetAppInfo {
"propagateDependencies" = ($manifest.PSObject.Properties.Name -eq 'PropagateDependencies') -and $manifest.PropagateDependencies
"dependencies" = @(if($manifest.PSObject.Properties.Name -eq 'dependencies'){$manifest.dependencies | ForEach-Object { if ($_.PSObject.Properties.Name -eq 'id') { $id = $_.id } else { $id = $_.AppId }; @{ "id" = $id; "name" = $_.name; "publisher" = $_.publisher; "version" = $_.version }}})
}
Write-Host " (succeeded using altool)"
}
else {
if (!$assembliesAdded) {
Expand Down Expand Up @@ -1158,7 +1155,6 @@ function GetAppInfo {
"propagateDependencies" = $manifest.PropagateDependencies
}
$packageStream.Close()
Write-Host " (succeeded using codeanalysis)"
}
if ($cacheAppInfoPath) {
$appInfoCache | Add-Member -MemberType NoteProperty -Name $relativePath -Value $appInfo
Expand All @@ -1183,7 +1179,6 @@ function GetAppInfo {
}
}
catch [System.Reflection.ReflectionTypeLoadException] {
Write-Host " (failed)"
if ($_.Exception.LoaderExceptions) {
$_.Exception.LoaderExceptions | Select-Object -Property Message | Select-Object -Unique | ForEach-Object {
Write-Host "LoaderException: $($_.Message)"
Expand All @@ -1200,7 +1195,6 @@ function GetAppInfo {
}
Pop-Location
}
Write-GroupEnd
}

function GetLatestAlLanguageExtensionVersionAndUrl {
Expand Down Expand Up @@ -1635,4 +1629,12 @@ function GetHostOs {
}
}
return $hostOs
}

function Write-PSCallStack {
Param(
[string] $message
)
Write-Host "PS CallStack $message :"
Get-PSCallStack | ForEach-Object { Write-Host "- $($_.FunctionName) ($([System.IO.Path]::GetFileName($_.ScriptName)) Line $($_.ScriptLineNumber))" }
}
12 changes: 10 additions & 2 deletions ObjectHandling/Import-TestToolkitToNavContainer.ps1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@
Only Test Runner and Test Framework are/will be available in the online Business Central environment
.Parameter environment
Environment in which you want to import test toolkit.
.Parameter appSymbolsFolder
Folder where the app symbols should be stored when using compilerfolder
.Example
Import-TestToolkitToBcContainer -containerName test2
.Example
Expand Down Expand Up @@ -68,8 +70,8 @@ function Import-TestToolkitToBcContainer {
[switch] $useDevEndpoint,
[hashtable] $replaceDependencies = $null,
[Hashtable] $bcAuthContext,
[string] $environment

[string] $environment,
[string] $appSymbolsFolder
)

$telemetryScope = InitTelemetryScope `
Expand Down Expand Up @@ -121,6 +123,12 @@ try {
}
Write-Host -ForegroundColor Green "TestToolkit successfully published"
}
elseif ($compilerFolder) {
$appFiles = GetTestToolkitApps -compilerFolder $compilerFolder -includeTestRunnerOnly:$includeTestRunnerOnly -includeTestFrameworkOnly:$includeTestFrameworkOnly -includeTestLibrariesOnly:$includeTestLibrariesOnly -includePerformanceToolkit:$includePerformanceToolkit
$appFiles | ForEach-Object {
Copy-Item -Path $_ -Destination $appSymbolsFolder -Force
}
}
else {
$inspect = docker inspect $containerName | ConvertFrom-Json
if ($inspect.Config.Labels.psobject.Properties.Match('maintainer').Count -eq 0 -or $inspect.Config.Labels.maintainer -ne "Dynamics SMB") {
Expand Down
5 changes: 5 additions & 0 deletions ReleaseNotes.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ Issue 1303 from AL-Go repository - renew federated token when access token needs
Issue 3590 EarliestMatching select option added to Download-BcNuGetPackageToFolder
Regression when from PR 3760 - Push-BcNuGetPackage doesn't work on PowerSHell 5
Issue 3772 Get-BcNuGetPackageId name length limit
Two new parameter in Sort-AppFoldersByDependencies - selectSubordinates and SkippedApps. If specified, then only the appFolders specified in the first array plus the apps depending on these recursively will be sorted. The remaining folder not included in the sorted appfolders list will be returned in skippedApps.
Remove some excessive logging
If Import-TestToolkitToBcContainer was running with useCompilerFolder, then it would require a container to be present at that time.
Make Run-AlPipeline do just-in-time creation of CompilerFolder and Containers
Issue 1338 from AL-Go repository: Multi-Project Repository Test App Dependencies Not Resolved

6.0.29
Issue 3591 When using Publish-NAVApp to publish an app, which fails compilation in the service, the command might hang forever - the fix for this is a temporary hack put in place for the versions which doesn't work.
Expand Down

0 comments on commit 7c8e804

Please sign in to comment.