Frontier Hackathon 2023

2023 Frontier Hackathon

The Hackathon is described here.

Timeline

• Thursday June 22, 11:00 EST: Brief meeting to define expectations, tasks; also to help everyone get started compiling and running, and understand the machine.

Zoom link here.

• Friday June 23, 13:00 to 14:00 EST: Preparation session by OLCF. Not everybody needs to attend, but those not attending will need to catch up later.

Zoom link here.

• Wednesday June 28 - Friday June 30, 11:00 EST to 17:00 EST: Workshop.

Zoom link here.

CarpetX Participants

• Steve Brandt

• Lorenzo Ennoggi

• Roland Haas

• Liwei Ji

• Jay Kalinani

• Lucas Timotheo Sanches

• Erik Schnetter

• John Holmen: OLCF POC

• Weiqun Zhang: AMReX support

Communication channel:

Slack Workspace

Please join the Frontier Hackathon - June 2023 Slack workspace using the following link: https://join.slack.com/t/frontier-hack-2023-06/shared_invite/zt-1wwlh59sg-1XgbThZmmkTnxByejw0aoQ

After joining the workspace, please search for and join the #team-asterx channel.

Compiling and running on Crusher

Erik's Simfactory settings for Crusher are here. (Erik is busy this week and doesn't have time to clean this up – these files should be moved into a proper repository, possibly Simfactory itself, after some testing by others.)

In addition to using these settings, you need to use the eschnett/crusher branch of the flesh and CarpetX.

Compiling and running on Frontier

Create ET folder in the home directory:

cd ~/
mkdir ET
cd ET

Download the code via the following commands:

curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/master/GetComponents
chmod a+x GetComponents
./GetComponents --root Cactus --parallel --no-shallow https://raw.githubusercontent.com/jaykalinani/AsterX/frontier/Docs/thornlist/asterx.th

#Note that the above thornlist (located in frontier branch of AsterX repository) downloads flesh and CarpetX from the branch eschnett/crusher

Simfactory files for Frontier are available in the folder:

Cactus/repos/AsterX/Docs/compile-notes/frontier

Copy frontier.ini to Cactus/simfactory/mdb/machines/.

Copy frontier.cfg to Cactus/simfactory/mdb/optionlists/.

Copy frontier.run to Cactus/simfactory/mdb/runscripts/.

Copy frontier.sub to Cactus/simfactory/mdb/submitscripts/.

Copy defs.local.ini to Cactus/simfactory/etc/., and edit user account details, source and base directory paths accordingly.

Return to Cactus directory and compile using the following command:

./simfactory/bin/sim build -j16 <config_name> --thornlist=./thornlists/asterx.th --machine=frontier

Example command to create-submit a job via simfactory

./simfactory/bin/sim submit B1 --parfile=./arrangements/AsterX/AsterX/test/Balsara1_shocktube.par --config=<config_name> --machine=frontier --allocation=ast182 --procs=64 --num-threads=1 --ppn-used=8 --queue=batch --walltime 00:01:00

Pre-workshop meeting notes

• Erik's notes

Quick links

• Frontier User Guide: https://docs.olcf.ornl.gov/systems/frontier_user_guide.html
• AsterX Team introduction slides here

Tasks

Profiling

Jay

Debugging

To-do list

• Compile newest version of AMReX (23.06)
• Use -munsafe-fp-atomics
• Test ROCm 5.2

HPCToolkit

• Cheatsheet here
• Getting started on Frontier here
• To output measurements, in the runscript, prefix the call to the executable with:

    hpcrun                                      \
    -o hpctoolkit-measurements                  \
    -e CPUTIME                                  \
    -e gpu=amd                                  \
    -t                                          \
``