DQN Global routing torch

In this repo, I reimplement dqn global routing in pytorch, and make the Q-network deeper to get better training results.

Differences from original implementation

• deeper network
  The original implementation uses only three hidden layers, this implementation uses a deeper architecture similar to resnet (in Trainer/algos/agent/dqn.py) for better performance.
• remove memory burn-in
  The original implemenation used A* memory burn-in to make the training faster, this implementation didn't used memory burn-in, but can still achieve 0.75 wire-length winning rate against A* algorithm. (with the help of deeper network)
• pretrain stage You can pretrain the agent to avoid training agent from scratch everytime in evaluation stage.

How to run?

Step0. Install packages

pip install -r requirements.txt

Step1. Generate Dataset

To train the agent, first generate the train and eval dataset

# train data
python gen_data.py --benchmarkNumber 30 --gridSize 8 --netNum 20 --vCap 4  --hCap 4 --maxPinNum 5 --reducedCapNum 3 --prefix ./train_data_/
# test data
python gen_data.py --benchmarkNumber 20 --gridSize 8 --netNum 20 --vCap 4  --hCap 4 --maxPinNum 5 --reducedCapNum 3 --prefix ./test_data_/

Step2. start training

default options is in configs.yaml (or Trainer/Router.py main_fn), you can modify it in yaml or overwrite it through command line interface

command line (or yaml) args

• mode:str = 'train' || 'eval'
• algos:str ( algos names are the filenames in Trainer/algos/agent folder)
  • dqn, dtqn, rainbow_dqn
• wandbName:str
• hid_layer:int = 3
• emb_dim:int = 64
• context_len:int = 5
• early_stop:bool = False
• result_dir:str = solutionsDRL
• load_ckpt:bool
• save_ckpt:bool
• self_play_episode_num:int -> self play budget
• enable_wandb:bool=True -> use wandb for plotting
• data_folder:str -> netlist data to solve

python main.py --config_path train.yaml --wandbName demo

Step3. start eval

eval run the 20 test benchmark, each with 150 self-play number (in configs.yaml), eval will take longer time to run (about 1hr on a RTX3060 GPU)

python main.py --config_path test.yaml --wandbName demo

Step4. Go to wandb site to check the result

Step5. plot wire len

cd eval
python Evaluation.py 
python VisualizeResults.py 

will generate wirelen images in ./eval/VisualizeResult.

Compare dqn wire length with A* algorithm (0.75 winning rate) (lower is better)

Without pretrain model

Evalutaion with out pretrain model will get similar result, compare with evalutation with pretrain model (~=0.75 winning rate), but will takes a longer time to train.

# fist delete the pretrain model -->  model/dqn.ckpt
rm model/dqn.ckpt
# train from scratch in eval stage
python main.py --config_path test.yaml --wandbName without_pretrain

wire len without pretrain

Training time

Training time on a Nvidia 3060 GPU

• without pretrain 110min
• pretrain 60min

Network depth

The success rate increase if we use deeper network.

### original implementation dont use res block
python main.py --config_path train.yaml --wandbName original_3_layer   --hid_layer 0
python main.py --config_path test.yaml  --wandbName original_3_layer   --self_play_episode_num 50 --hid_layer 0
#  first delete ckpt in model/dqn.ckpt
rm ./model/dqn.ckpt
### 3 resblock  agent
python main.py --config_path train.yaml --hid_layer 3  --wandbName _3_resblock_10_layer
python main.py --config_path test.yaml --self_play_episode_num 50 --hid_layer 3 --wandbName _3_resblock_10_layer

other algorithms

DTQN

To run Deep Transformer Q-Networks , change the algos option through command line.

## pretrain
python main.py --config_path train.yaml --algos dtqn --run_benchmark_num 30
## eval
python main.py --config_path test.yaml  --algos dtqn --self_play_episode_num 150

DTQN result

dtqn has similar results compare to dqn.

compare dtqn wire length with A* algorithm (0.7 winning rate)