o 40h.@sddlmZddlmZddlmZddlmZmZeZ ddZ dd Z d d Z ddlmZdd l Z d dZddlmZdeedddfddZd S)) Translator)slugify) BeautifulSoup)GENERAL_META_DESCRIPTIONGENERAL_META_KEYWORDScCs,z t|d}|jWSty|YSw)NEnglish) translator translateresult Exception)text translationr;/var/www/html/mimamsha/mimansha/apps/mimansha_main/utils.pytranslate_to_englishs   rcCs"t|d}dd|dD}|S)z Extracts all superscript text enclosed in tags from an HTML string. Args: html_text (str): The HTML content to parse Returns: list: A list of all superscript texts found in order of appearance html.parsercSsg|]}|qSr)get_text).0suprrr sz(extract_superscripts..r)rfind_all) html_textsoup superscriptsrrrextract_superscriptss rcCsddl}ddlm}|jdd||jd}|jdd||jd}dD]}|jd |d d ||jd}|jd |d d ||jd}q |jdd ||jd}|dd|}||}|dd|}|dd|}d dd|D}|}|S)a Convert HTML content to plain text by removing HTML tags and decoding HTML entities. Args: html_content (str): HTML content to convert Returns: str: Plain text without HTML tags and with decoded HTML entities rN)unescapez flagsz ) divph1h2h3h4h5h6librtrz<\s*z[^>]*> z<\s*/\s*z\s*>z <\s*br\s*/?>z<[^>]*>z + z\n{3,}z css|]}|VqdS)N)strip)rlinerrr Fsz%html_to_plain_text..) rehtmlrsubDOTALL IGNORECASEjoin splitlinesr-) html_contentr0rtagrrrhtml_to_plain_text!s r9Nc Cstd|tj}|s|ddS|d}i}td|tjtjBD]}|d}|d}||t|<q#|s^d}t||tjtjB}|D]}|d}|d}||t|<qI|s|d} | D]} t d | }|r|d}|d}||t|<qgtj dd|tjd } t | d } | j d d D]'} | j r| j jdvrqt dddt| }|t| krt |d }| |qt| }d}t|D]\}}|d|d|d|d|d 7}q|d7}||dS)ad Processes HTML content with numbered references in [1] format and creates a footnotes section. Formats reference numbers as superscript with specific styling. Args: html_content (str): HTML string containing references like [1] and footnotes in the format: [footnotes] [1] Footnote text... [2] Another footnote... [/footnotes] Returns: dict: { 'processed_content': HTML with linked superscript footnotes, 'footnotes_template': Formatted footnotes section with back-to-text links } z!\[footnotes\](.*?)\[\/footnotes\]r)processed_contentfootnotes_templaterz)\[(\d+)\]\s*(.*?)(?=\n\[\d+\]|\Z|\n\s*\n)z\[(\d+)\](.*?)(?=\n\[\d+\]|\Z)r+z \[(\d+)\](.*)rrT)string)scriptstylez \[(\d+)\]cSs*d|dd|dd|ddS)Nzz )group)mrrrs*z#process_footnotes..z

References

z
[zi]
z
a
)r0searchr3r@r-finditer MULTILINEintsplitmatchr2rrparentnamestr replace_withsorteditems)r7footnotes_blockfootnotes_text footnotesrHnumr patternmatcheslinesr.processed_htmlr text_nodenew_textnew_soupr:r;rrrprocess_footnotesPsr           lrZ)settingsMIMAMSHAz/static/icons/MIMAMSHA.pngzhttps://www.mimamsha.comcKsnid|d|d|ddddd|d |d |d |d dd dd|d|d|dddddd|S)N meta_titlemeta_description meta_keywordsauthorr\og_typewebsite canonical_urlog_titleog_descriptionog_image site_name twitter_cardsummary_large_image twitter_titletwitter_description twitter_image twitter_sitez @mimamshatwitter_creatorrobotsz index, followr)r]r^r_rfrlrckwargsrrrget_meta_contents-sJ      rq) translatepyrdjango.utils.textrbs4r stopwordsrrr rrr9r0rZ django.confr[rqrrrrs(    , \