Align NPU to CPU #2560

KodiaqQ · 2024-03-08T11:07:11Z

Changes

Aligned NPU configuration to CPU for 8bit quantization;

Reason for changes

NPU supports CPU configuration;
CPU configuration as the base;
Removed unused configurations (ConvolutionBackpropData, GroupConvolutionBackpropData);

Related tickets

132512

Tests

TBD

codecov · 2024-03-08T11:09:42Z

Codecov Report

Attention: Patch coverage is 8.33333% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 29.94%. Comparing base (ac15a65) to head (e5cfe06).
Report is 1 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #2560       +/-   ##
============================================
- Coverage    62.07%   29.94%   -32.13%     
============================================
  Files          494      494               
  Lines        45775    45791       +16     
============================================
- Hits         28413    13714    -14699     
- Misses       17362    32077    +14715

Files	Coverage Δ
nncf/common/utils/dot_file_rw.py	`58.33% <0.00%> (-31.95%)`	⬇️
nncf/quantization/algorithms/min_max/algorithm.py	`20.17% <9.09%> (-76.10%)`	⬇️

... and 262 files with indirect coverage changes

Flag	Coverage Δ
COMMON	`?`
ONNX	`?`
OPENVINO	`?`
TENSORFLOW	`29.94% <8.33%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
common	`76.35% <0.00%> (-12.08%)`	⬇️
torch	`0.01% <ø> (-32.85%)`	⬇️
tensorflow	`93.74% <ø> (ø)`
onnx	`0.00% <ø> (-93.07%)`	⬇️
openvino	`0.00% <ø> (-94.23%)`	⬇️
ptq	`15.24% <9.09%> (-65.68%)`	⬇️

KodiaqQ · 2024-03-08T11:19:09Z

Linters fails due to #2561

ljaljushkin · 2024-03-11T12:21:17Z

Should NPU config also contain this?

            "attributes": {
                "scales": "unified"
            },

ljaljushkin · 2024-03-11T12:22:05Z

Also noticed this difference

KodiaqQ · 2024-03-11T12:26:38Z

Should NPU config also contain this?

            "attributes": {
                "scales": "unified"
            },

Maybe @alexsu52 and @AlexKoff88 can answer this question.

KodiaqQ · 2024-03-11T12:27:43Z

Also noticed this difference

What should we do with this? Are these parameters used for NPU?

alexsu52 · 2024-03-12T06:24:35Z

Should NPU config also contain this?
            "attributes": {
                "scales": "unified"
            },
Maybe @alexsu52 and @AlexKoff88 can answer this question.

Good question. Scale unification should be applied only for operation from CPU hardware config to align behavior between CPU and NPU in INT8 quantization. What affects other precision, we should not change their behavior.

alexsu52

Do you have results of the conformance test for NPU config?

Please think about what test needs to be added to check the equality of the cpu, gpu and npu configs for int8.

nncf/common/hardware/configs/cpu.json

KodiaqQ · 2024-03-13T07:47:03Z

Do you have results of the conformance test for NPU config?

Please think about what test needs to be added to check the equality of the cpu, gpu and npu configs for int8.

We have no tests for NPU in the conformance as well. I haven't run these tests.

ljaljushkin · 2024-03-14T20:34:21Z

tests/torch/data/reference_graphs/quantized/adjust_paddings/npu_all_activations_int8_requnt.dot

-"15 /nncf_model_output_1" [id=15, label="nncf_model_output_#15", style=filled, type=nncf_model_output];
-"16 /nncf_model_output_2" [id=16, label="nncf_model_output_#16", style=filled, type=nncf_model_output];
-"17 /nncf_model_output_3" [id=17, label="nncf_model_output_#17", style=filled, type=nncf_model_output];
+"6 MultiBranchesModel/MaxPool2d[max_pool_b]/SymmetricQuantizer/symmetric_quantize_0" [color=green, id=6, label="AFQ_[B:8 M:S SGN:U PC:N]_#6_G0", style=filled, type=symmetric_quantize];


previously conv had asymmetric int8 activations

Currently NPU config have symmetric activations on the first place, that's why there're 2 FQ.

The same for all requant tests

Other references just changed assymetric to symmetric in 8bit activations

Thanks for the clarification.
But why do we have an asymmetric quantizer after the relu layer?

AlexKoff88 · 2024-03-18T11:17:57Z

Just for the record, the main motivation for keeping the config is QAT for NPU which has some custom features such as W4A4 support.

tests/common/test_hardware_config.py

.github/workflows/nightly.yml

tests/torch/data/reference_graphs/quantized/hw/NPU/inception_v3.dot

KodiaqQ · 2024-05-03T12:36:58Z

nncf/quantization/algorithms/min_max/algorithm.py

@@ -206,6 +206,7 @@ def __init__(
            else:
                self._preset = QuantizationPreset.PERFORMANCE

+        self._override_device()


@alexsu52, I've added overriding. Review, please.

KodiaqQ · 2024-05-06T08:30:56Z

@alexsu52, please, review

alexsu52

LGTM

github-actions bot added the NNCF Common Pull request that updates NNCF Common label Mar 8, 2024

KodiaqQ marked this pull request as ready for review March 8, 2024 11:18

KodiaqQ requested a review from a team as a code owner March 8, 2024 11:18

KodiaqQ requested review from alexsu52 and ljaljushkin March 8, 2024 11:19

KodiaqQ force-pushed the nm/align_npu_with_cpu branch from 9d4ec97 to 4e6bbf9 Compare March 8, 2024 14:00

KodiaqQ marked this pull request as draft March 11, 2024 10:08

KodiaqQ marked this pull request as ready for review March 11, 2024 16:10

github-actions bot added NNCF TF Pull requests that updates NNCF TensorFlow NNCF PT Pull requests that updates NNCF PyTorch labels Mar 11, 2024

KodiaqQ requested a review from AlexKoff88 March 11, 2024 16:16

alexsu52 reviewed Mar 13, 2024

View reviewed changes

nncf/common/hardware/configs/cpu.json Show resolved Hide resolved

KodiaqQ requested a review from alexsu52 March 14, 2024 12:43

ljaljushkin reviewed Mar 14, 2024

View reviewed changes

ljaljushkin approved these changes Mar 14, 2024

View reviewed changes

AlexKoff88 approved these changes Mar 18, 2024

View reviewed changes

KodiaqQ self-assigned this Mar 19, 2024

github-actions bot added dependencies Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF PTQ Pull requests that updates NNCF PTQ labels Mar 20, 2024

KodiaqQ commented Mar 20, 2024

View reviewed changes

tests/common/test_hardware_config.py Show resolved Hide resolved

KodiaqQ assigned alexsu52 Mar 21, 2024

alexsu52 reviewed Mar 22, 2024

View reviewed changes

.github/workflows/nightly.yml Outdated Show resolved Hide resolved

tests/torch/data/reference_graphs/quantized/hw/NPU/inception_v3.dot Show resolved Hide resolved

KodiaqQ marked this pull request as draft April 2, 2024 07:27

KodiaqQ force-pushed the nm/align_npu_with_cpu branch from b741e0b to 6dd4caa Compare April 2, 2024 08:00

github-actions bot removed dependencies Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF PTQ Pull requests that updates NNCF PTQ labels Apr 2, 2024

KodiaqQ requested a review from alexsu52 April 2, 2024 08:21

KodiaqQ marked this pull request as ready for review April 2, 2024 08:21

KodiaqQ force-pushed the nm/align_npu_with_cpu branch from 1e959af to 54c317e Compare April 16, 2024 17:08

KodiaqQ added 13 commits May 3, 2024 11:01

Align NPU to CPU

6896dc9

Update Torch refs

8212142

Fix minors

c04d5dd

Update TF refs

fde712b

Update reference

3e107b5

Change refs

75cbd70

Align NPU to CPU

675d354

Fix minors

76edc63

Update reference

432fa33

Fix refs

7aaa789

Added test & missed config

92825ad

Fixed NPU config

bab6136

Remove tests for PT

9cfbde0

KodiaqQ force-pushed the nm/align_npu_with_cpu branch from 54c317e to e5cfe06 Compare May 3, 2024 11:53

github-actions bot added the NNCF PTQ Pull requests that updates NNCF PTQ label May 3, 2024

Change NPU to CPU in PTQ

e5cfe06

KodiaqQ commented May 3, 2024

View reviewed changes

alexsu52 approved these changes May 6, 2024

View reviewed changes

alexsu52 merged commit b611322 into openvinotoolkit:develop May 6, 2024
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Align NPU to CPU #2560

Align NPU to CPU #2560

KodiaqQ commented Mar 8, 2024 •

edited

Loading

codecov bot commented Mar 8, 2024 •

edited

Loading

KodiaqQ commented Mar 8, 2024

ljaljushkin commented Mar 11, 2024

ljaljushkin commented Mar 11, 2024

KodiaqQ commented Mar 11, 2024

KodiaqQ commented Mar 11, 2024

alexsu52 commented Mar 12, 2024

alexsu52 left a comment

KodiaqQ commented Mar 13, 2024

ljaljushkin Mar 14, 2024

ljaljushkin Mar 14, 2024

ljaljushkin Mar 14, 2024

KodiaqQ Mar 15, 2024

AlexKoff88 commented Mar 18, 2024

KodiaqQ May 3, 2024

KodiaqQ commented May 6, 2024

alexsu52 left a comment

Align NPU to CPU #2560

Align NPU to CPU #2560

Conversation

KodiaqQ commented Mar 8, 2024 • edited Loading

Changes

Reason for changes

Related tickets

Tests

codecov bot commented Mar 8, 2024 • edited Loading

Codecov Report

KodiaqQ commented Mar 8, 2024

ljaljushkin commented Mar 11, 2024

ljaljushkin commented Mar 11, 2024

KodiaqQ commented Mar 11, 2024

KodiaqQ commented Mar 11, 2024

alexsu52 commented Mar 12, 2024

alexsu52 left a comment

Choose a reason for hiding this comment

KodiaqQ commented Mar 13, 2024

ljaljushkin Mar 14, 2024

Choose a reason for hiding this comment

ljaljushkin Mar 14, 2024

Choose a reason for hiding this comment

ljaljushkin Mar 14, 2024

Choose a reason for hiding this comment

KodiaqQ Mar 15, 2024

Choose a reason for hiding this comment

AlexKoff88 commented Mar 18, 2024

KodiaqQ May 3, 2024

Choose a reason for hiding this comment

KodiaqQ commented May 6, 2024

alexsu52 left a comment

Choose a reason for hiding this comment

KodiaqQ commented Mar 8, 2024 •

edited

Loading

codecov bot commented Mar 8, 2024 •

edited

Loading