o –hé ã@sZdZddlZddlZddlmZGdd„dejƒZGdd„dejƒZGdd „d ejƒZdS) z\ Based on https://pytorch.org/tutorials/beginner/transformer_tutorial.html#define-the-model éN)Únncs>eZdZdZ‡fdd„Zed dd„ƒZdd„Zd d „Z‡Z S) ÚSinusoidalEncodingz2 Uses sine & cosine to represent position cs"tƒ ¡| d| ||¡¡dS)NÚpe)ÚsuperÚ__init__Úregister_bufferÚbuild_position)ÚselfÚ model_dimÚmax_len©Ú __class__©úi/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/models/constituency/positional_encoding.pyrs zSinusoidalEncoding.__init__NcCs˜t |¡ d¡}t t d|d¡t d¡ |¡}t ||¡}t ||¡|dd…ddd…f<t ||¡|dd…ddd…f<|durJ|j |d}|S)NérégˆÃ@©Údevice) ÚtorchÚarangeÚ unsqueezeÚexpÚmathÚlogÚzerosÚsinÚcosÚto)r r rÚpositionÚdiv_termrrrrrs$    z!SinusoidalEncoding.build_positioncCs^t|ƒ|jjdkr*|jj}|jjd}| dd¡| d|j|t|ƒd|d¡|j|S)Nrrrr)ÚmaxrÚshaperrr)r Úxrr!rrrÚforwards     zSinusoidalEncoding.forwardcCs |jjdS)Nr)rr!)r rrrr *s zSinusoidalEncoding.max_len©N) Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚ staticmethodrr#r Ú __classcell__rrr rr s    rcs,eZdZdZd ‡fdd„ Zd dd„Z‡ZS) ÚAddSinusoidalEncodingz Uses sine & cosine to represent position. Adds the position to the given matrix Default behavior is batch_first éécótƒ ¡t||ƒ|_dSr$©rrrÚencoding©r Úd_modelr r rrr4ó zAddSinusoidalEncoding.__init__çð?cCsvt|jƒdkr | tj|jd|jd¡}| |jddd¡}nt|jƒdkr5| tj|jd|jd¡}|||S)z¼ Adds the positional encoding to the input tensor The tensor is expected to be of the shape B, N, D Properly masking the output tensor is up to the caller érrréÿÿÿÿr)Úlenr!r0rrrÚexpand)r r"ÚscaleÚtimingrrrr#8s  zAddSinusoidalEncoding.forward©r,r-)r4©r%r&r'r(rr#r*rrr rr+.sr+cs*eZdZdZd‡fdd„ Zdd„Z‡ZS) ÚConcatSinusoidalEncodingzŠ Uses sine & cosine to represent position. Concats the position and returns a larger object Default behavior is batch_first r,r-cr.r$r/r1r rrrMr3z!ConcatSinusoidalEncoding.__init__cCsrt|jƒdkr | tj|jd|jd¡}| |jddd¡}n| tj|jd|jd¡}tj||fdd}|S)Nr5rrrr6)Údim)r7r!r0rrrr8Úcat)r r"r:Úoutrrrr#Qs z ConcatSinusoidalEncoding.forwardr;r<rrr rr=Gsr=)r(rrrÚModulerr+r=rrrrÚs #