Neural Network Processing Unit Operator Support

This section summarizes neural network operators supported by the SyNAP VS6x0/SL16x0 class of NPUs and accompanying software stack. For each operator type, the supported tensor types and execution engines is also documented. Designing networks that maximise the use of operators executed in the NN core will provide the best performances.

Table 14 Legend

Acronym	Description
NN	Neural Network Engine
PPU	Parellel Processing Unit
TP	Tensor Processor
asym-u8	asymmetric affine uint8
asym-i8	asymmetric affine int8
pc-sym-i8	per channel symmetric int8
fp32	floating point 32 bits
fp16	floating point 16 bits
h	half
int16	int16
int32	int32

Note

int16 dynamic fixed point convolution is supported by NN Engine in their multiplication. Other layers follow the tables, if asym-u8 in not available in NN column, int16 is also not available.

Basic Operations

	Tensor Types			Execution Engine
Operator	Input	Kernel	Output	NN	TP	PPU
CONV2D	asym-u8	asym-u8	asym-u8	✓
	asym-i8	pc-sym-i8	asym-i8	✓		✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16			✓
CONV1D	asym-u8	asym-u8	asym-u8	✓
	asym-i8	pc-sym-i8	asym-i8	✓		✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16			✓
DECONVOLUTION	asym-u8	asym-u8	asym-u8	✓
	asym-i8	pc-sym-i8	asym-i8	✓		✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16			✓
DECONVOLUTION1D	asym-u8	asym-u8	asym-u8	✓
	asym-i8	pc-sym-i8	asym-i8	✓		✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16			✓
GROUPED_CONV2D	asym-u8	asym-u8	asym-u8	✓
	asym-i8	pc-sym-i8	asym-i8	✓		✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16			✓

Note

convolutions are executed in the NN emgine only if they satisfy the following conditions: stride == 1, kernel_size <= 15x15, dilation size + kernel size <= 15x15 If any of these conditions is not satisfied, the convolution will require support of the TP core and will run considerably slower

	Tensor Types			Execution Engine
Operator	Input	Kernel	Output	NN	TP	PPU
FULLY_CONNECTED	asym-u8	asym-u8	asym-u8		✓
	asym-i8	pc-sym-i8	asym-i8		✓
	fp32	fp32	fp32			✓

Activation Operations

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
ELU	asym-u8	asym-u8		✓	✓
	asym-i8	asym-i8		✓	✓
	fp32	fp32		✓	✓
	fp16	fp16		✓	✓
HARD_SIGMOID	asym-u8	asym-u8		✓	✓
	asym-i8	asym-i8		✓	✓
	fp32	fp32		✓	✓
	fp16	fp16		✓	✓
SWISH	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
LEAKY_RELU	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
PRELU	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
RELU	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
RELUN	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
RSQRT	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
SIGMOID	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
SOFTRELU	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
SQRT	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
TANH	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
ABS	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
CLIP	asym-u8	asym-u8		✓	✓
	asym-i8	asym-i8		✓	✓
	fp32	fp32			✓
	fp16	fp16		✓	✓
EXP	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
LOG	asym-u8	asym-u8		✓	✓
	asym-i8	asym-i8		✓	✓
	fp32	fp32			✓
	fp16	fp16		✓	✓
NEG	asym-u8	asym-u8		✓	✓
	asym-i8	asym-i8		✓	✓
	fp32	fp32			✓
	fp16	fp16		✓	✓
MISH	asym-u8	asym-u8		✓	✓
	asym-i8	asym-i8		✓	✓
	fp32	fp32			✓
	fp16	fp16		✓	✓

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
SOFTMAX	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
LOG_SOFTMAX	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
SQUARE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
SIN	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
LINEAR	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
ERF	asym-u8	asym-u8		✓	✓
	asym-i8	asym-i8		✓	✓
	fp32	fp32			✓
	fp16	fp16		✓	✓
GELU	asym-u8	asym-u8		✓	✓
	asym-i8	asym-i8		✓	✓
	fp32	fp32			✓
	fp16	fp16		✓	✓

Elementwise Operations

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
ADD	asym-u8	asym-u8	✓
	asym-i8	asym-i8	✓
	fp32	fp32			✓
	fp16	fp16			✓
SUBTRACT	asym-u8	asym-u8	✓
	asym-i8	asym-i8	✓
	fp32	fp32			✓
	fp16	fp16			✓
MULTIPLY	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
DIVIDE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
MAXIMUM	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
MINIMUM	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
POW	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
FLOORDIV	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
MATRIXMUL	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
RELATIONAL_OPS	asym-u8	bool8			✓
	asym-i8	bool8			✓
	fp32	bool8			✓
	fp16	bool8			✓
	bool8	bool8			✓
LOGICAL_OPS	bool8	bool8			✓
LOGICAL_NOT	bool8	bool8			✓
SELECT	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
	bool8	bool8			✓
ADDN	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓

Normalization Operations

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
BATCH_NORM	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
LRN2	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
L2_NORMALIZE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
LAYER_NORM	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
INSTANCE_NORM	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
BATCHNORM_SINGLE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
MOMENTS	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
	fp16	fp16			✓
GROUP_NORM	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓

Reshape Operations

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
EXPAND_BROADCAST	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
SLICE	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
SPLIT	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
CONCAT	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
STACK	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
UNSTACK	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
RESHAPE	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
SQUEEZE	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
PERMUTE	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
REORG	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
SPACE2DEPTH	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
DEPTH2SPACE	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
	bool8	bool8

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
BATCH2SPACE	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
SPACE2BATCH	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
PAD	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
REVERSE	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
STRIDED_SLICE	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
REDUCE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓

	Tensor Types		Execution Engine
Operator	Input	Output	NN	TP	PPU
ARGMAX	asym-u8	asym-u8 / int16 / int32			✓
	asym-i8	asym-u8 / int16 / int32			✓
	fp32	int32			✓
	fp16	asym-u8 / int16 / int32			✓
ARGMIN	asym-u8	asym-u8 / int16 / int32			✓
	asym-i8	asym-u8 / int16 / int32			✓
	fp32	int32			✓
	fp16	asym-u8 / int16 / int32			✓
SHUFFLECHANNEL	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓

RNN Operations

	Tensor Types			Execution Engine
Operator	Input	Kernel	Output	NN	TP	PPU
LSTMUNIT_OVXLIB	asym-u8	asym-u8	asym-u8		✓	✓
	asym-i8	pc-sym-i8	asym-i8		✓	✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16		✓	✓
CONV2D_LSTM	asym-u8	asym-u8	asym-u8	✓
	asym-i8	pc-sym-i8	asym-i8	✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16			✓
CONV2D_LSTM_CELL	asym-u8	asym-u8	asym-u8	✓
	asym-i8	pc-sym-i8	asym-i8	✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16			✓
LSTM_OVXLIB	asym-u8	asym-u8	asym-u8		✓	✓
	asym-i8	pc-sym-i8	asym-i8		✓	✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16		✓	✓

	Tensor Types			Execution Engine
Operator	Input	Kernel	Output	NN	TP	PPU
GRUCELL_OVXLIB	asym-u8	asym-u8	asym-u8		✓	✓
	asym-i8	pc-sym-i8	asym-i8		✓	✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16		✓	✓
GRU_OVXLIB	asym-u8	asym-u8	asym-u8		✓	✓
	asym-i8	pc-sym-i8	asym-i8		✓	✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16		✓	✓
SVDF	asym-u8	asym-u8	asym-u8		✓	✓
	asym-i8	pc-sym-i8	asym-i8		✓	✓
	fp32	fp32	fp32			✓
	fp16	fp16	fp16		✓	✓

Pooling Operations

	Tensor Types			Execution Engine
Operator	Input	Output	NN	TP	PPU
POOL	asym-u8	asym-u8	✓	✓
	asym-i8	asym-i8	✓	✓
	fp32	fp32			✓
	fp16	fp16		✓
ROI_POOL	asym-u8	asym-u8		✓	✓
	asym-i8	asym-i8		✓	✓
	fp32	fp32			✓
	fp16	fp16		✓	✓
POOLWITHARGMAX	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
UPSAMPLE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓

Miscellaneous Operations

	Tensor Types			Execution Engine
Operator	Input	Output	NN	TP	PPU
PROPOSAL	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
VARIABLE	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
DROPOUT	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
RESIZE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
DATACONVERT	asym-u8	asym-u8		✓
	asym-i8	asym-i8		✓
	fp32	fp32			✓
	fp16	fp16		✓
FLOOR	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓

	Tensor Types			Execution Engine
Operator	Input	Output	NN	TP	PPU
EMBEDDING_LOOKUP	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
GATHER	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
GATHER_ND	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
SCATTER_ND	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
GATHER_ND_UPDATE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
TILE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓

	Tensor Types			Execution Engine
Operator	Input	Output	NN	TP	PPU
ELTWISEMAX	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
SIGNAL_FRAME	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
CONCATSHIFT	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
UPSAMPLESCALE	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp16	fp16			✓
ROUND	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
CEIL	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓

	Tensor Types			Execution Engine
Operator	Input	Output	NN	TP	PPU
SEQUENCE_MASK	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
REPEAT	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
ONE_HOT	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓
CAST	asym-u8	asym-u8			✓
	asym-i8	asym-i8			✓
	fp32	fp32			✓
	fp16	fp16			✓