Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(common): unify pull-request creation scripts #9888

Merged
merged 2 commits into from
Oct 30, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 120 additions & 0 deletions resources/build/ci/pull-requests.inc.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
#
# Commit and upload files as a PR to s.keyman.com, help.keyman.com, or other
# repositories
#

# Defaults for CI pull requests
# TODO: this may belong better somewhere else
if [[ -z ${KEYMAN_PR_USERNAME+x} ]]; then
KEYMAN_PR_USERNAME="Keyman Build Server"
fi

if [[ -z ${KEYMAN_PR_EMAIL+x} ]]; then
KEYMAN_PR_EMAIL="[email protected]"
fi

# Add files to the target repository. Saves and restores PWD
#
# Usage:
# ci_add_files repo_path pathspec...
# Parameters:
# 1: repo_path local path to root of the target git repository
# 2: pathspec... one or more paths to add
# Requires:
# * git
#
function ci_add_files {
local repo="$1"
shift
pushd "$repo" >/dev/null
# Git will return an error if the pathspec does not exist, but will not return
# an error if the pathspec exists but there are no new files to add. This is
# the behaviour we want.
git add "$*" || builder_die "git returned error $? when attempting to add $*"
popd >/dev/null
}

# Test if there are any changes to the git cache in the target repository.
# Used in conjunction with `ci_add_files` to verify that files have or have
# not been added, depending on the scenario
#
# Usage:
# if ci_repo_has_cached_changes repo_path; then ... fi
# Parameters:
# 1: repo_path local path to root of the target git repository
# Requires:
# * git
#
function ci_repo_has_cached_changes {
local repo="$1"
local return_code=1
pushd "$repo" >/dev/null
if ! git diff --cached --no-ext-diff --quiet --exit-code; then
return_code=0
fi
popd >/dev/null
return $return_code
}

# Opens a pull request on the target repository, first committing cached files
# to a new branch (with a hopefully-unique branch name). git vars user.name and
# user.email will be configured if not already present. Adds the label 'auto'
# to the pull request.
#
# Usage:
# ci_open_pull_request repo branch_base commit_message
# Parameters:
# 1: repo_path local path to root of the target git repository
# 2: branch_base branch name to use (a unique value `/<unique>` will be
# appended)
# 3: commit_message commit message and pull request title to use
# Requires:
# * git
# * hub
# * uuidgen (or TeamCity `$BUILD_NUMBER` if `uuidgen` not present on system)
# Example:
# ci_open_pull_request "$S_KEYMAN_COM" auto/keymanweb/release "auto: KeymanWeb release $VERSION_RELEASE""
#
function ci_open_pull_request {
local repo="$1"
local branch_base="$2"
local commit_message="$3"
pushd "$repo" >/dev/null

if ! git config user.name > /dev/null; then
git config user.name "$KEYMAN_PR_USERNAME"
fi
if ! git config user.email > /dev/null; then
git config user.email "$KEYMAN_PR_EMAIL"
fi

# We want a unique branch name, so we append either a random or the TeamCity
# build number
local uuid=
if [[ -z ${BUILD_NUMBER+x} ]]; then
uuid=$(uuidgen)
else
uuid=TC-$BUILD_NUMBER
fi

local branch="$branch_base/$uuid"
local current_branch="$(git branch --show-current)"

builder_echo "Creating new branch '$branch' on '$repo'"
git switch -c "$branch"
builder_echo "$commit_message"
git commit -m "$commit_message"
git push origin "$branch"
builder_echo "Push complete"

hub pull-request -f --no-edit -l auto
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's "probably fine" for our CI agents, so long as we note the requirement... just noting that this assumes hub is available. Is that documented somewhere?

Asking because I don't have (or need) hub installed in my personal dev-environment, and I don't think it's listed in our dev prerequisite list. Not sure where we'd document extra resources needed for the agents.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, given this is just for CI. Note that this is listed in the reqs in the comments for the script.

builder_echo "Pull request created"

git switch "$current_branch"
builder_echo "Switched back to $current_branch"

popd >/dev/null

builder_echo "Pull request successfully created"
return 0
}
48 changes: 48 additions & 0 deletions resources/build/ci/test/test-pull-requests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
#
# Tests the pull-requests.inc.sh functions. This is not really intended to be
# used as an automated test, but rather only to be run manually after changes to
# pull-requests.inc.sh.
#

## START STANDARD BUILD SCRIPT INCLUDE
# adjust relative paths as necessary
THIS_SCRIPT="$(readlink -f "${BASH_SOURCE[0]}")"
. "${THIS_SCRIPT%/*}/../../../../resources/build/build-utils.sh"
# END STANDARD BUILD SCRIPT INCLUDE

cd "$THIS_SCRIPT_PATH"

. "$KEYMAN_ROOT/resources/build/ci/pull-requests.inc.sh"

test_repo_base=`mktemp -d`
test_repo="$test_repo_base/ci-test-repo"
test_filename="$(uuidgen).md"

builder_echo "Preparing local repository clone in $test_repo"

pushd "$test_repo_base" >/dev/null
git clone https://github.com/keymanapp/ci-test-repo
popd >/dev/null

builder_echo "Creating test file $test_filename"

echo "This is a test $test_filename" > "$test_repo/$test_filename"

builder_echo blue "Testing ci_add_files"

ci_add_files "$test_repo" .

builder_echo blue "Testing ci_repo_has_cached_changes"

if ! ci_repo_has_cached_changes "$test_repo"; then
builder_die "Expected to have a file to commit!"
fi

builder_echo blue "Testing ci_open_pull_request"

ci_open_pull_request "$test_repo" "test/ci" "chore: testing CI"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So... does the test actually make a real pull request to the destination site (ci-test-repo) when run and leave it up? No cleanup for the results of this call?

Granted, I suppose that might be why ci-test-repo is there - to prevent pollution of our production repos with test PRs of this type?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. And yes. You can even look at said repo and see my test PRs (now manually cleaned up). Given the script is designed for manual run, I figured that was okay.


rm -rf "$test_repo_base"

builder_echo green "Tests passed"
63 changes: 10 additions & 53 deletions resources/build/help-keyman-com.sh
Original file line number Diff line number Diff line change
@@ -1,13 +1,4 @@
#!/usr/bin/env bash

#
# Prevents 'clear' on exit of mingw64 bash shell
#
SHLVL=0

set -e
set -u

## START STANDARD BUILD SCRIPT INCLUDE
# adjust relative paths as necessary
THIS_SCRIPT="$(readlink -f "${BASH_SOURCE[0]}")"
Expand Down Expand Up @@ -135,49 +126,6 @@ function upload_keyman_help {
cp -rv "$helppath"/* "$dstpath/"
}

#
# Commit and push to the help.keyman.com repo
# Creates a pull request with the 'auto' label
# Which a GitHub action will watch for in order
# to automatically process and merge it
#

function commit_and_push {
echo "Committing and pushing updated Keyman for $platform documentation"

pushd $HELP_KEYMAN_COM

if [ ! -z "${TEAMCITY_VERSION-}" ]; then
git config user.name "Keyman Build Server"
git config user.email "[email protected]"
fi

local branchname="auto/$platform-help-$VERSION_WITH_TAG"
local modifiedfiles=$(help_product_path)

local basebranch="master"

git checkout -b $branchname $basebranch
git add $modifiedfiles || return 1
git diff --cached --no-ext-diff --quiet --exit-code && {
# if no changes then don't do anything.
echo "No changes to commit"
popd
return 0
}

echo "changes added to cache...>>>"
local commitmessage="auto: Keyman for $platform help deployment"
git commit -m "$commitmessage" || return 1
git push origin $branchname || return 1
hub pull-request -b $basebranch -l auto -m "$commitmessage" || return 1
popd

echo "Push to help.keyman.com complete"

return 0
}

#
# Main
#
Expand Down Expand Up @@ -213,4 +161,13 @@ generate_markdown_help || exit 1
echo "Uploading Keyman for $platform documentation to help.keyman.com"

upload_keyman_help || exit 1
commit_and_push || exit 1

ci_add_files "$HELP_KEYMAN_COM" "$(help_product_path)"
if ! ci_repo_has_cached_changes "$HELP_KEYMAN_COM"; then
echo "No changes to commit"
exit 0
fi

ci_open_pull_request "$HELP_KEYMAN_COM" "auto/$platform-help-$VERSION_WITH_TAG" "auto: Keyman for $platform help deployment"

exit 0
10 changes: 6 additions & 4 deletions resources/builder.inc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -845,6 +845,8 @@ builder_describe() {
description="$(builder_trim "${sub[@]:1}")"
fi

local original_value="$value"

if [[ $value =~ ^: ]]; then
# Parameter is a target
local target_path=
Expand Down Expand Up @@ -939,9 +941,9 @@ builder_describe() {
fi

if [[ -z "${description}" ]]; then
description=$(_builder_get_default_description "$value")
description=$(_builder_get_default_description "$original_value")
fi
_builder_params[${value}]="$description"
_builder_params[${original_value}]="$description"

shift
done
Expand Down Expand Up @@ -1482,7 +1484,7 @@ builder_display_usage() {
echo "Actions: "

for e in "${_builder_actions[@]}"; do
if _builder_item_in_glob_array "$e" "${_builder_params[@]}"; then
if [[ ! -z "${_builder_params[$e]+x}" ]]; then
description="${_builder_params[$e]}"
else
description=$(_builder_get_default_description "$e")
Expand All @@ -1494,7 +1496,7 @@ builder_display_usage() {
echo "Targets: "

for e in "${_builder_targets[@]}"; do
if _builder_item_in_glob_array "$e" "${_builder_params[@]}"; then
if [[ ! -z "${_builder_params[$e]+x}" ]]; then
description="${_builder_params[$e]}"
else
description=$(_builder_get_default_description "$e")
Expand Down
24 changes: 15 additions & 9 deletions web/ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
#

# set -x
set -eu

## START STANDARD BUILD SCRIPT INCLUDE
# adjust relative paths as necessary
Expand All @@ -14,8 +13,7 @@ THIS_SCRIPT="$(readlink -f "${BASH_SOURCE[0]}")"
## END STANDARD BUILD SCRIPT INCLUDE

. "$KEYMAN_ROOT/resources/shellHelperFunctions.sh"

WORKING_DIRECTORY=`pwd`
. "$KEYMAN_ROOT/resources/build/ci/pull-requests.inc.sh"

# This script runs from its own folder
cd "$THIS_SCRIPT_PATH"
Expand All @@ -24,15 +22,15 @@ cd "$THIS_SCRIPT_PATH"

S_KEYMAN_COM=

builder_describe "Defines and implements the CI build steps for Keyman Engine for Web (KMW)." \
builder_describe "CI processes for Keyman Engine for Web releases (KMW)." \
"@/web/src/tools/building/sourcemap-root prepare:s.keyman.com" \
"build" \
"test Runs all unit tests." \
"post-test Runs post-test cleanup. Should be run even if a prior step fails." \
"test Runs all unit tests" \
"post-test Runs post-test cleanup. Should be run even if a prior step fails" \
"validate-size Runs the build-size comparison check" \
"prepare Prepare upload artifacts for specified target(s)" \
":s.keyman.com Target: builds artifacts for s.keyman.com " \
":downloads.keyman.com Target: builds artifacts for downloads.keyman.com" \
":s.keyman.com Builds artifacts for s.keyman.com" \
":downloads.keyman.com Builds artifacts for downloads.keyman.com" \
"--s.keyman.com=S_KEYMAN_COM Sets the root location of a checked-out s.keyman.com repo"

builder_parse "$@"
Expand Down Expand Up @@ -151,7 +149,15 @@ if builder_start_action prepare:s.keyman.com; then
node "$KEYMAN_ROOT/web/build/tools/building/sourcemap-root/index.js" null "$sourcemap" --sourceRoot "https://s.keyman.com/kmw/engine/$VERSION/src"
done

# Actual construction of the PR will be left to CI-config scripting for now.
# Construct the PR
echo "Committing and pushing KeymanWeb release $VERSION to s.keyman.com"

ci_add_files "$S_KEYMAN_COM" "kmw/engine/$VERSION"
if ! ci_repo_has_cached_changes "$S_KEYMAN_COM"; then
builder_die "No release was added to s.keyman.com, something went wrong"
fi

ci_open_pull_request "$S_KEYMAN_COM" auto/keymanweb/release "auto: KeymanWeb release $VERSION"

builder_finish_action success prepare:s.keyman.com
fi
Expand Down
Loading