o 'h0@s.ddlmZddlZddlmZddlmZddlmZddl mZ ddl m Z m Z mZmZmZddlmZdd lmZdd lmZmZmZmZmZd8ddZd9ddZed8ddZed8ddZeedd:ddZ d;dd(d)Z"d?d-d.Z#d@d/d0Z$ed1d dAdBd6d7Z%dS)C) annotationsN)IncrementalDecoder)Counter) lru_cache) FREQUENCIESKO_NAMESLANGUAGE_SUPPORTED_COUNTTOO_SMALL_SEQUENCEZH_NAMES) is_suspiciously_successive_range)CoherenceMatches)is_accentuatedis_latinis_multi_byte_encodingis_unicode_range_secondary unicode_range iana_namestrreturn list[str]cst|rtdtd|j}|dd}idtddD]/}|t|g}|rNt|}|dur4qt |d urN|vrBd|<|d 7<d 7qt fd d DS) zF Return associated unicode ranges in a single byte code page. z.Function not supported on multi-byte code pagez encodings.ignore)errorsr@NFrcs g|] }|dkr|qS)g333333?).0character_rangecharacter_count seen_rangesrQ/var/www/html/olx_land/venv/lib/python3.10/site-packages/charset_normalizer/cd.py 8s z*encoding_unicode_range..) rOSError importlib import_modulerrangedecodebytesrrsorted)rdecoderpichunkrrrr!encoding_unicode_ranges.   r. primary_rangecCs>g}tD]\}}|D]}t||kr||nq q|S)z> Return inferred languages used with a unicode range. )ritemsrappend)r/ languageslanguage characters characterrrr!unicode_range_languages@s  r6cCs<t|}d}|D] }d|vr|}nq|durdgSt|S)z Single-byte encoding language association. Some code page are heavily linked to particular language(s). This function does the correspondence. NLatin Latin Based)r.r6)runicode_rangesr/specified_rangerrr!encoding_languagesOsr;cCs`|ds|ds|ds|dkrdgS|ds|tvr"dgS|ds+|tvr.d gSgS) z Multi-byte encoding language association. Some code page are heavily linked to particular language(s). This function does the correspondence. shift_ iso2022_jpeuc_jcp932JapanesegbChinese iso2022_krKorean) startswithr r)rrrr!mb_encoding_languagescsrF)maxsizer3tuple[bool, bool]cCsBd}d}t|D]}|st|rd}|rt|durd}q||fS)zg Determine main aspects from a supported language if it contains accents and if is pure Latin. FT)rrr)r3target_have_accentstarget_pure_latinr5rrr!get_target_featuresxs  rKFr4ignore_non_latinboolc sg}tddD}tD]6\}}t|\}}|r |dur q|dur'|r'qt|}tfdd|D} | |} | dkrE||| fqt|ddd d }d d|DS) zE Return associated languages associated to given characters. css|]}t|VqdSN)r)rr5rrr! z%alphabet_languages..Fcsg|]}|vr|qSrr)rcr4rr!r"sz&alphabet_languages..g?cS|dSNrrxrrr!z$alphabet_languages..TkeyreversecSsg|]}|dqS)rr)rcompatible_languagerrr!r")anyrr0rKlenr1r)) r4rLr2source_have_accentsr3language_charactersrIrJrcharacter_match_countratiorrRr!alphabet_languagess$   rdordered_charactersfloatcCs|tvr t|dd}tt|}t|}tt|}|dk}t|td|D]\}}||vr2q)t||} ||} t|| } |durPt| | dkrPq)|durct| | |dkrc|d7}q)t|d| } t|| d } |d|}||d }tt|t| @}tt|t| @}t| dkr|dkr|d7}q)t| dkr|dkr|d7}q)|t| d ks|t| d kr|d7}q)q)|t|S) aN Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language. The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit). Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.) z not availablerFTrNg?) r ValueErrorsetr_zipr&indexintabs)r3recharacter_approved_countFREQUENCIES_language_setordered_characters_count target_language_characters_countlarge_alphabetr5character_rankcharacter_rank_in_languageexpected_projection_ratiocharacter_rank_projectioncharacters_before_sourcecharacters_after_sourcecharacters_beforecharacters_afterbefore_match_countafter_match_countrrr!characters_popularity_comparesf         rdecoded_sequencecCsi}|D]?}|dur qt|}|durqd}|D] }t||dur'|}nq|dur.|}||vr9|||<q|||7<qt|S)a Given a decoded text sequence, return a list of str. Unicode range / alphabet separation. Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list; One containing the latin letters and the other hebrew. FN)isalpharr lowerlistvalues)rlayersr5rlayer_target_rangediscovered_rangerrr!alpha_unicode_splits,   rresultslist[CoherenceMatches]r csfi|D]}|D]}|\}}|vr|g|<q||qqfddD}t|ddddS)z This function merge results previously given by the function coherence_ratio. The return type is the same as coherence_ratio. cs.g|]}|tt|t|dfqS)rh)roundsumr_)rr3per_language_ratiosrr!r"1sz*merge_coherence_ratios..cSrSrTrrUrrr!rW<rXz(merge_coherence_ratios..TrY)r1r))rresult sub_resultr3rcmergerrr!merge_coherence_ratios#s   rcst|D]}|\}}|dd}|vrg|<||qtfddDr@g}D] }||t|fq0|S|S)u We shall NOT return "English—" in CoherenceMatches because it is an alternative of "English". This function only keeps the best match and remove the em-dash in it. u—c3s |] }t|dkVqdS)rN)r_)re index_resultsrr!rOOsz/filter_alt_coherence_matches..)dictreplacer1r^max)rrr3rc no_em_namefiltered_resultsrrr!filter_alt_coherence_matches?s ri皙? threshold lg_inclusion str | NonecCsg}d}d}|dur|dng}d|vrd}|dt|D]K}t|}|} tdd| D} | tkr8q d d | D} |pEt| |D]$} t| | } | |krRqF| d krZ|d 7}| | t | d f|dkrjnqFq t t |ddddS)z Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers. A layer = Character extraction by alphabets/ranges. FrN,r8Tcss|]\}}|VqdSrNrrrQorrr!rOqrPz"coherence_ratio..cSsg|]\}}|qSrrrrrr!r"vr]z#coherence_ratio..g?rrhricSrSrTrrUrrr!rWrXz!coherence_ratio..rY) splitremoverr most_commonrr rdrr1rr)r)rrrrrLsufficient_match_countlg_inclusion_listlayersequence_frequenciesrrpopular_character_orderedr3rcrrr!coherence_ratioZs@   r)rrrr)r/rrr)r3rrrH)F)r4rrLrMrr)r3rrerrrf)rrrr)rrrr )rr rr )rN)rrrrfrrrr )& __future__rr$codecsr collectionsr functoolsrtyping TypeCounterconstantrrr r r mdr modelsr utilsrrrrrr.r6r;rFrKrdrrrrrrrrr!s6        %    # P '