Prepare for v1.1 release step #2 (#36)

* decouple torch dependency in code
* refine integration test
* update nni==2.5
* add doc for proxylessNAS

Co-authored-by: Li Lyna Zhang <[email protected]>

.github/workflows/integration-test.yml

@@ -28,16 +28,14 @@ jobs:
         path: |
           ~/.nn_meter
           /home/runner/work/nn-Meter/data/testmodels
-          /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages
-        key: Data-${{hashFiles('nn_meter/configs/predictors.yaml')}}-Test-${{hashFiles('tests/integration_test.py')}}
+        key: ${{hashFiles('nn_meter/configs/predictors.yaml')}}-${{hashFiles('tests/integration_test.py')}}
 
     - name: Install dependencies
-      if: steps.cache.outputs.cache-hit != 'true'
       run: |
         pip install tensorflow==1.15.0
         pip install onnx==1.9.0
-        pip install torch==1.7.1
-        pip install torchvision==0.8.2
+        pip install torch==1.9.0
+        pip install torchvision==0.10.0
         pip install onnx-simplifier
         
     - name: Install nn-Meter
@@ -47,5 +45,8 @@ jobs:
       run: python tests/integration_test.py
 
     - name: Diff result with reference
-      run: diff tests/test_result.txt tests/reference_result.txt
+      run: diff tests/data/reference_result.txt tests/test_result.txt
+
+    - name: clean env
+      run: rm tests/test_result.txt  
 

.github/workflows/integration-test_nni_based_torch.yml

@@ -28,20 +28,13 @@ jobs:
         path: |
           ~/.nn_meter
           /home/runner/work/nn-Meter/data/testmodels
-          /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages
-        key: Data-${{hashFiles('nn_meter/configs/predictors.yaml')}}-Test-${{hashFiles('tests/integration_test.py')}}
+        key: ${{hashFiles('nn_meter/configs/predictors.yaml')}}-${{hashFiles('tests/integration_test.py')}}
 
     - name: Install dependencies
-      if: steps.cache.outputs.cache-hit != 'true'
       run: |
-        pip install tensorflow==1.15.0
-        pip install onnx==1.9.0
-        pip install torch==1.7.1
-        pip install torchvision==0.8.2
-        pip install onnx-simplifier
-    
-    - name: Install nni
-      run: pip install nni==2.4
+        pip install torch==1.9.0
+        pip install torchvision==0.10.0
+        pip install nni==2.5
 
     - name: Install nn-Meter
       run: pip install -U .
@@ -50,5 +43,7 @@ jobs:
       run: python tests/integration_test_torch.py --apply-nni
 
     - name: Diff result with reference
-      run: diff tests/reference_result_nni_based_torch.txt tests/test_result_nni_based_torch.txt
+      run: diff tests/data/reference_result_nni_based_torch.txt tests/test_result_nni_based_torch.txt
 
+    - name: clean env
+      run: rm tests/test_result_nni_based_torch.txt

.github/workflows/integration-test_onnx_based_torch.yml

@@ -28,16 +28,13 @@ jobs:
         path: |
           ~/.nn_meter
           /home/runner/work/nn-Meter/data/testmodels
-          /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages
-        key: Data-${{hashFiles('nn_meter/configs/predictors.yaml')}}-Test-${{hashFiles('tests/integration_test.py')}}
+        key: ${{hashFiles('nn_meter/configs/predictors.yaml')}}-${{hashFiles('tests/integration_test.py')}}
 
     - name: Install dependencies
-      if: steps.cache.outputs.cache-hit != 'true'
       run: |
-        pip install tensorflow==1.15.0
         pip install onnx==1.9.0
-        pip install torch==1.7.1
-        pip install torchvision==0.8.2
+        pip install torch==1.9.0
+        pip install torchvision==0.10.0
         pip install onnx-simplifier
         
     - name: Install nn-Meter
@@ -47,5 +44,7 @@ jobs:
       run: python tests/integration_test_torch.py --apply-onnx
 
     - name: Diff result with reference
-      run: diff tests/reference_result_onnx_based_torch.txt tests/test_result_onnx_based_torch.txt
-
+      run: diff tests/data/reference_result_onnx_based_torch.txt tests/test_result_onnx_based_torch.txt
+
+    - name: clean env
+      run: rm tests/test_result_onnx_based_torch.txt

README.md

@@ -1,4 +1,4 @@
-Note: This is an alpha (preview) version which is still under refining.
+Note: This is an beta (preview) version which is still under refining.
 
 **nn-Meter** is a novel and efficient system to accurately predict the inference latency of DNN models on diverse edge devices. The key idea is dividing a whole model inference into kernels, i.e., the execution units of fused operators on a device, and conduct kernel-level prediction. We currently evaluate four popular platforms on a large dataset of 26k models. It achieves 99.0% (mobile CPU), 99.1% (mobile Adreno 640 GPU), 99.0% (mobile Adreno 630 GPU), and 83.4% (Intel VPU) prediction accuracy.
 
@@ -63,7 +63,7 @@ The stable version of wheel binary package will be released soon.
 
 To apply for hardware latency prediction, nn-Meter provides two types of interfaces：
 
-- command line `nn-meter` after `nn-meter`[installation](QuickStart.md#Installation).
+- command line `nn-meter` after `nn-meter` [installation](QuickStart.md#Installation).
 - Python binding provided by the module `nn_meter`
 
 Here is a summary of supported inputs of the two methods.
@@ -180,8 +180,9 @@ We release the dataset, and provide an interface of `nn_meter.dataset` for users
 
 To empower affordable DNN on the edge and mobile devices, hardware-aware NAS searches both high accuracy and low latency models. In particular, the search algorithm only considers the models within the target latency constraints during the search process.
 
-Currently we provides example of end-to-end [multi-trial NAS](https://nni.readthedocs.io/en/stable/NAS/multi_trial_nas.html), which is a [random search algorithm](https://arxiv.org/abs/1902.07638) on [SPOS NAS](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123610528.pdf) search space. More examples of more hardware-aware NAS and model compression algorithms are coming soon.
+Currently we provide two examples of hardware-aware NAS, including end-to-end [multi-trial NAS](https://nni.readthedocs.io/en/stable/NAS/multi_trial_nas.html) which is a [random search algorithm](https://arxiv.org/abs/1902.07638) on [SPOS NAS](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123610528.pdf) search space, and the popular [ProxylessNAS](https://nni.readthedocs.io/en/stable/NAS/Proxylessnas.html), which is a one-shot NAS algorithm with hardware-efficient loss function. More examples of other widely-used hardware-aware NAS and model compression algorithms are coming soon.
 
+### Multi-trial SPOS Demo
 To run multi-trail SPOS demo, NNI should be installed through source code by following [NNI Doc](https://nni.readthedocs.io/en/stable/Tutorial/InstallationLinux.html#installation)
 
 ```bash
@@ -194,7 +195,7 @@ Then run multi-trail SPOS demo:
 python ${NNI_ROOT}/examples/nas/oneshot/spos/multi_trial.py
 ```
 
-### How the demo works
+#### How the demo works
 
 Refer to [NNI Doc](https://nni.readthedocs.io/en/stable/nas.html) for how to perform NAS by NNI.
 
@@ -221,6 +222,20 @@ exp.run(exp_config, port)
 
 In `exp_config`, `dummy_input` is required for tracing shape info.
 
+### ProxylessNAS Demo
+
+To run the one-shot ProxylessNAS demo, users can run the NNI ProxylessNAS training demo:
+
+```bash
+python ${NNI_ROOT}/examples/nas/oneshot/proxylessnas/main.py --applied_hardware <hardware> --reference_latency <reference latency (ms)>
+```
+
+#### How the demo works
+
+Refer to [NNI Doc](https://nni.readthedocs.io/en/stable/nas.html) for how to perform NAS by NNI.
+
+ProxylessNAS currently builds a lookup table, that stores the measured latency of each candidate building block in the search space. The latency sum of all building blocks in a candidate model will be treated as the model inference latency. With leveraging nn-Meter in NNI, users can apply ProxylessNAS to search efficient DNN models on more types of edge devices. In NNI implementation, a `HardwareLatencyEstimator` predicts expected latency for the mixed operation based on the path weight of `ProxylessLayerChoice`. To call nn-Meter in NNI ProxylessNAS, users can add the arguments of "`--applied_hardware <hardware> --reference_latency <reference latency (ms)>`" in the [example](https://github.com/microsoft/nni/blob/master/examples/nas/oneshot/proxylessnas/main.py).
+
 ## Bench Dataset
 
 To evaluate the effectiveness of a prediction model on an arbitrary DNN model, we need a representative dataset that covers a large prediction scope. nn-Meter collects and generates 26k CNN models. (Please refer the paper for the dataset generation method.)

docs/hardware-aware-model-design.md

@@ -20,7 +20,9 @@ Our HW-NAS framework firstly automatically selects the hardware-friendly operato
 
 Besides the search space specialization, our HW-NAS framework also allows combining nn-Meter with existing NAS algorithms in the optimization objectives and constraints. As described in [[4]](http://arxiv.org/abs/2101.09336), the HW-NAS algorithms often consider hardware efficiency metrics as the constraints of existing NAS formulation or part of the scalarized loss functions (e.g., the loss is weighted sum of both cross entropy loss and hardware-aware penalty). Since the NAS process may sample up to millions of candidate model architectures, the obtaining of hardware metrics must be accurate and efficient.
 
-nn-Meter is now integrated with [NNI](https://github.com/microsoft/nni), the AutoML framework also published by Microsoft, and could be combined with existing NAS algorithms seamlessly. [This doc](https://nni.readthedocs.io/en/stable/NAS/multi_trial_nas.html) show how to construct a latency constraint filter in [random search algorithm](https://arxiv.org/abs/1902.07638) on [SPOS NAS](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123610528.pdf) search space. Users could use this filter in multiple phases of the NAS process, e.g., the architecture searching phase and the super-net training phase.
+nn-Meter is now integrated with [NNI](https://github.com/microsoft/nni), the AutoML framework also published by Microsoft, and could be combined with existing NAS algorithms seamlessly. [This doc](https://nni.readthedocs.io/en/stable/NAS/HardwareAwareNAS.html#endtoend-multi-trial-spos-demo) show how to construct a latency constraint filter in [random search algorithm](https://arxiv.org/abs/1902.07638) on [SPOS NAS](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123610528.pdf) search space. Users could use this filter in multiple phases of the NAS process, e.g., the architecture searching phase and the super-net training phase. 
+
+Another example is [ProxylessNAS](https://arxiv.org/pdf/1812.00332.pdf), a hardware-aware one-shot NAS algorithm. ProxylessNAS applies the expected latency of the model to build a differentiable metric and design efficient neural network architectures for hardware. The latency loss is added as a regularization term for architecture parameter optimization. The [official implementation](https://github.com/mit-han-lab/ProxylessNAS) of ProxylessNAS supports different targeted hardware, including 'mobile', 'cpu', 'gpu8', 'flops'. In the [current implementation](https://nni.readthedocs.io/en/stable/NAS/Proxylessnas.html) on NNI, users could use a latency estimator based on nn-Meter to predict expected latency for the mixed operation on other types of mobile and edge hardware. By using nn-Meter, please specifying the arguments of `--applied_hardware <hardware> --reference_latency <reference latency (ms)>` in the [example](https://github.com/microsoft/nni/blob/master/examples/nas/oneshot/proxylessnas/main.py).
 
 ***Note that current nn-Meter project is limited to the latency prediction. For the other hardware metrics, e.g., energy consumption is another important metric in edge computing. Collaborations and contributions together with nn-Meter are highly welcomed!***
 

docs/overview.md

@@ -1,5 +1,5 @@
 # Overview
-Note: This is an alpha (preview) version which is still under refining.
+Note: This is an beta (preview) version which is still under refining.
 
 nn-Meter is a novel and efficient system to accurately predict the inference latency of DNN models on diverse edge devices.
 

docs/quick_start.md

@@ -38,7 +38,7 @@ The stable version of wheel binary package will be released soon.
 ## "Hello World" example on torch model
 nn-Meter is an accurate inference latency predictor for DNN models on diverse edge devices. nn-Meter supports tensorflow pb-file, onnx file, torch model and nni IR model for latency prediction.
 
-Here is an example script to predict latency for Resnet18 in torch. To run the example, package `torch`, `torchvision`, `onnx` and `onnx-simplifier` are required. The well tested versions are `torch==1.7.1`, `torchvision==0.8.2`, `onnx==1.9.0` and `onnx-simplifier==0.3.6`.   
+Here is an example script to predict latency for Resnet18 in torch. To run the example, package `torch`, `torchvision`, `onnx` and `onnx-simplifier` are required. The well tested versions are `torch==1.9.0`, `torchvision==0.10.0`, `onnx==1.9.0` and `onnx-simplifier==0.3.6`.   
 
 ```python
 from nn_meter import load_latency_predictor

nn_meter/__init__.py

@@ -18,7 +18,7 @@
 )
 from .utils.utils import download_from_url
 from .prediction import latency_metrics
-from .dataset import bench_dataset # TODO: add GNNDataloader and GNNDataset here @wenxuan
+from .dataset import bench_dataset
 import logging
 from functools import partial, partialmethod
 

nn_meter/dataset/__init__.py

@@ -1,4 +1,3 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT license.
 from .bench_dataset import bench_dataset
-from .gnn_dataloader import GNNDataset, GNNDataloader

nn_meter/dataset/gnn_dataloader.py

@@ -42,7 +42,6 @@ def __init__(self, train=True, device="cortexA76cpu_tflite21", split_ratio=0.8):
         err_str = "Not supported device type"
         assert device in hws, err_str
         self.device = device
-        self.data_dir = __user_dataset_folder__
         self.train = train
         self.split_ratio = split_ratio
         self.adjs = {}
@@ -54,34 +53,25 @@ def __init__(self, train=True, device="cortexA76cpu_tflite21", split_ratio=0.8):
         self.raw_data = {}
         self.name_list = []
         self.latencies = {}
-        self.download_data()
+        self.data_dir = bench_dataset(data_folder=__user_dataset_folder__)
         self.load_model_archs_and_latencies(self.data_dir)
         self.construct_attrs()
-        self.name_list = list(
-            filter(lambda x: x in self.latencies, self.name_list))
-
-    def download_data(self):
-        datasets = bench_dataset()
+        self.name_list = list(filter(lambda x: x in self.latencies, self.name_list))
 
     def load_model_archs_and_latencies(self, data_dir):
-        filelist = os.listdir(data_dir)
-        for filename in filelist:
-            if os.path.splitext(filename)[-1] != '.jsonl':
-                continue
-            self.load_model(os.path.join(data_dir, filename))
+        for filename in data_dir:
+            self.load_model(filename)
 
     def load_model(self, fpath):
         """
         Load a concrete model type.
         """
-        # print('Loading models in ', fpath)
         assert os.path.exists(fpath), '{} does not exists'.format(fpath)
 
         with jsonlines.open(fpath) as reader:
             _names = []
             for obj in reader:
                 if obj[self.device]:
-                    # print(obj['id'])
                     _names.append(obj['id'])
                     self.latencies[obj['id']] = float(obj[self.device])
 
@@ -245,7 +235,6 @@ def construct_graphs(self):
         for gid in range(self.length):
             (adj, attrs), latency, op_types = self.dataset[gid]
             u, v = torch.nonzero(adj, as_tuple=True)
-            # import pdb; pdb.set_trace()
             graph = dgl.graph((u, v))
             MAX_NORM = torch.tensor([1]*len(op_types) + [6963, 6963, 224, 224, 11, 4])
             attrs = attrs / MAX_NORM
@@ -272,4 +261,4 @@ def __next__(self):
         batch_indexes = self.indexes[start:end]
         batch_graphs = [self.graphs[i] for i in batch_indexes]
         batch_latencies = [self.latencies[i] for i in batch_indexes]
-        return torch.tensor(batch_latencies), dgl.batch(batch_graphs)
+        return torch.tensor(batch_latencies), dgl.batch(batch_graphs)

nn_meter/ir_converters/utils.py

@@ -60,7 +60,7 @@ def model_file_to_graph(filename: str, model_type: str, input_shape=(1, 3, 224,
             'inception_v3': 'models.inception_v3()',
             'googlenet': 'models.googlenet()',
             'shufflenet_v2': 'models.shufflenet_v2_x1_0()',
-            'mobilenet_v2': 'models.mobilenet_v2()',  # noqa: F841
+            'mobilenet_v2': 'models.mobilenet_v2()',
             'resnext50_32x4d': 'models.resnext50_32x4d()',
             'wide_resnet50_2': 'models.wide_resnet50_2()',
             'mnasnet': 'models.mnasnet1_0()',
@@ -69,7 +69,7 @@ def model_file_to_graph(filename: str, model_type: str, input_shape=(1, 3, 224,
             model = eval(torchvision_zoo_dict[filename])
         else:
             suppost_list = ", ".join([k for k in torchvision_zoo_dict])
-            raise ValueError(f"Unsupported model name in torchvision. Supporting list: {suppost_list}")
+            raise ValueError(f"Unsupported model name: {filename} in torchvision. Supporting list: {suppost_list}")
         return torch_model_to_graph(model, input_shape, apply_nni)
 
     else:

nn_meter/utils/utils.py

@@ -41,7 +41,7 @@ def try_import_onnx(require_version = ["1.9.0"]):
         require_version = [require_version]
     try:
         import onnx
-        if version.parse(onnx.__version__).release not in [version.parse(v).release for v in require_version]:
+        if version.parse(onnx.__version__).release[:2] not in [version.parse(v).release[:2] for v in require_version]:
             logging.warning(f'onnx=={onnx.__version__} is not well tested now, well tested version: onnx=={", ".join(require_version)}' )
         return onnx
     except ImportError:
@@ -53,19 +53,19 @@ def try_import_torch(require_version = ["1.9.0", "1.7.1"]):
         require_version = [require_version]
     try:
         import torch
-        if version.parse(torch.__version__).release not in [version.parse(v).release for v in require_version]:
+        if version.parse(torch.__version__).release[:2] not in [version.parse(v).release[:2] for v in require_version]:
             logging.warning(f'torch=={torch.__version__} is not well tested now, well tested version: torch=={", ".join(require_version)}' )
         return torch
     except ImportError:
         logging.error(f'You have not install the torch package, please install torch=={require_version[0]} and try again.')
         exit()
 
-def try_import_tensorflow(require_version = ["1.15.0"]):
+def try_import_tensorflow(require_version = ["2.6.0", "1.15.0"]):
     if isinstance(require_version, str):
         require_version = [require_version]
     try:
         import tensorflow
-        if version.parse(tensorflow.__version__).release not in [version.parse(v).release for v in require_version]:
+        if version.parse(tensorflow.__version__).release[:2] not in [version.parse(v).release[:2] for v in require_version]:
             logging.warning(f'tensorflow=={tensorflow.__version__} is not well tested now, well tested version: tensorflow=={", ".join(require_version)}' )
         return tensorflow
     except ImportError:
@@ -77,7 +77,7 @@ def try_import_nni(require_version = ["2.4", "2.5"]):
         require_version = [require_version]
     try:
         import nni
-        if version.parse(nni.__version__).release not in [version.parse(v).release for v in require_version]:
+        if version.parse(nni.__version__).release[:2] not in [version.parse(v).release[:2] for v in require_version]:
             logging.warning(f'nni=={nni.__version__} is not well tested now, well tested version: nni=={", ".join(require_version)}' )
         return nni
     except ImportError: