add transformer support?(matmul, layernorm)

[bigprince97]

No description provided.

[bigprince97]

Hello, this is very convenient repo to convert pytorch model to tensorrt, but it likely support cv well, support nlp poolly,I try to convert gpt2 model to tensorRT, but there are two operations that transformers(nlp very popular model, it is the base block to build gpt2, bert) needs don't support, include matmul, layernorm, c++ api support it, consider support it in python api? I am very expected to deploy gpt2 in tensorRT.

[bigprince97]

I convert gpt2 model to tensorRT successfully use this repo and some operation I write by myself, and the speed has been increased three times, thanks this good repo.

[mowayao]

I convert gpt2 model to tensorRT successfully use this repo and some operation I write by myself, and the speed has been increased three times, thanks this good repo.

Can you share you impl of matmul, plz?

[czs1886]

I convert gpt2 model to tensorRT successfully use this repo and some operation I write by myself, and the speed has been increased three times, thanks this good repo.

Can you share you impl of matmul, plz?

Just use addMatrixMultiply() which is defined here
You need to create a new converter in converters directory. You can copy mul.py and change its addElementwise to addMatrixMultiply.

[q248953144]

I convert gpt2 model to tensorRT successfully use this repo and some operation I write by myself, and the speed has been increased three times, thanks this good repo.

can you share your ways

[Fan9]

@bigprince97 can you shart your code? thanks!

[bigprince97]

Sorry to see the reply now， I write code last year, it can't run because of the update of the repo. fastertransformer supports gpt2 acceleration，It has pytorch interface， you can try.

[francescotaioli]

for anyone looking for the matmul, here's the code

@tensorrt_converter('torch.Tensor.__matmul__')
def convert_mul(ctx):
    input_a = ctx.method_args[0]
    input_b = ctx.method_args[1]
    output = ctx.method_return
    input_a_trt, input_b_trt = add_missing_trt_tensors(ctx.network, [input_a, input_b])
    input_a_trt, input_b_trt = broadcast_trt_tensors(ctx.network, [input_a_trt, input_b_trt], len(output.shape) - 1)
    
    #layer = ctx.network.add_elementwise(input_a_trt, input_b_trt, trt.ElementWiseOperation.PROD)
    layer = ctx.network.add_matrix_multiply(input_a_trt, trt.MatrixOperation.NONE, input_b_trt, trt.MatrixOperation.NONE)
    output._trt = layer.get_output(0)

Documentation for python api: here and here