R\w1@sddlmZmZmZddlmZddlZddlZddlm Z m Z m Z m Z ddlm Z mZddlmZddlmZydd lmZWnek reZYnXydd lmZWn(ek r Gd d d eZYnXed de DZedde DZedde DZeeddgBZejdZeddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3g Zejd4Z iZ!Gd5d6d6Z"dd7d7d8d9Z#Gd:d;d;Z$Gd<d=d=e$Z%Gd>d?d?e&Z'Gd@dAdAeZ(GdBdCdCeZ)dDdEZ*dS)F)absolute_importdivisionunicode_literals) text_typeN)EOFspaceCharacters asciiLettersasciiUppercase) encodingsReparseException)utils)StringIO)BytesIO)BufferedIOBasec@seZdZdS)rN)__name__ __module__ __qualname__rrC/var/www/dbchiro/venv/build/pip/pip/_vendor/html5lib/inputstream.pyrs rcCsg|]}|jdqS)ascii)encode).0itemrrr s rcCsg|]}|jdqS)r)r)rrrrrrs cCsg|]}|jdqS)r)r)rrrrrrs >.)sumr )r"rrrr(SszBufferedStream._bufferedBytescCsJ|jj|}|jj||jdd7 Normalized stream from source for use by html5lib. source can be either a file-object, local filename or a string. The optional encoding parameter must be a string that indicates the encoding. If specified, that encoding will be used, regardless of any BOM or later declaration (such as in a meta element) parseMeta - Look for a element containing encoding information u􏿿ru [-]u0([-](?![-])|(?s z5HTMLUnicodeInputStream.charsUntil..z^%sz[%s]+N)charsUntilRegExKeyErrorrjr)r5rLrMryr&rVrUendr2re) r" charactersoppositecharsr}regexr7mrrrrr charsUntil0s2   &    z!HTMLUnicodeInputStream.charsUntilcCso|dk rk|jdkr=||j|_|jd7_qk|jd8_|j|j|ksktndS)Nrr)rVr&rUr))r"rfrrrunget_s  zHTMLUnicodeInputStream.unget)rrrr<rir#rTrRrbr!rfrerJrOrrrrrrr@s  !    (  /r@c@syeZdZdZdddddZddZdd Zddd d Zd d ZddZ ddZ dS)rAzProvides a unicode stream of characters to the HTMLTokenizer. This class takes care of character encoding and removing or replacing incorrect byte-sequences and also provides column and line tracking. NTcCs|j||_tj||jt|df|_d|_d|_d|_|jddkr|j |||_n|j dS)aInitialises the HTMLInputStream. HTMLInputStream(source, [encoding]) -> Normalized stream from source for use by html5lib. source can be either a file-object, local filename or a string. The optional encoding parameter must be a string that indicates the encoding. If specified, that encoding will be used, regardless of any BOM or later declaration (such as in a meta element) parseMeta - Look for a element containing encoding information rIidz windows-1252rN) rR rawStreamr@r# codecNamerQ numBytesMetanumBytesChardetdefaultEncodingdetectEncodingrT)r"rBrCrDrErrrr#xs   zHTMLBinaryInputStream.__init__cCs6tj|jd|jd|_tj|dS)Nrrl)codecs getreaderrQrrSr@rT)r"rrrrTs zHTMLBinaryInputStream.resetc CsVt|dr|}n t|}y|j|jWnt|}YnX|S)zvProduces a file object from source. source can be either a file object, local filename or a string. r0)r=rr,r'r)r"rBrrrrrRs  z HTMLBinaryInputStream.openStreamc Cs|j}d}|dkr9|r9|j}d}n|dkr:|r:d}yyddlm}Wn"tk rddlm}YnXg}|}x[|js|jj|j }t |t st |sPn|j ||j|qW|j|jd}|jjdWq:tk r6Yq:Xn|dkrXd}|j}nidd6} |j| kr| |j}n||fS)NrI tentativer)UniversalDetectorrCz windows-1252z iso-8859-1) detectBOMdetectEncodingMetacharade.universaldetectorr ImportErrorchardet.universaldetectordonerr0rr>r/r)r2feedcloseresultr,rlower) r"rDrErC confidencerbuffersdetectorr encodingSubrrrrsB             z$HTMLBinaryInputStream.detectEncodingcCs|jddkstt|}|d kr:d}n|dkrJdS||jdkrv|jddf|_nF|jjd|j|df|_td|jd|fdS) NrrIutf-16 utf-16-be utf-16-lezutf-8rzEncoding changed from %s to %s)rrr)rQr)rrr,rTr )r" newEncodingrrrchangeEncodings     z$HTMLBinaryInputStream.changeEncodingcCsidtj6dtj6dtj6dtj6dtj6}|jjd}t|t s_t |j |dd}d}|s|j |}d}|s|j |dd }d }qn|jj |r|pd |S) zAttempts to detect at BOM at the start of the stream. If an encoding can be determined from the BOM return the name of the encoding otherwise return Nonezutf-8z utf-16-lez utf-16-bez utf-32-lez utf-32-beNrqr) rBOM_UTF8 BOM_UTF16_LE BOM_UTF16_BE BOM_UTF32_LE BOM_UTF32_BErr0r>r/r)getr,)r"bomDictstringrCr,rrrrs   zHTMLBinaryInputStream.detectBOMcCsk|jj|j}t|ts*tt|}|jjd|j}|dkrgd}n|S)z9Report the encoding declared by the meta element rutf-16 utf-16-be utf-16-lezutf-8)rrr) rr0rr>r/r)EncodingParserr, getEncoding)r"r parserrCrrrrs    z(HTMLBinaryInputStream.detectEncodingMeta) rrrr<r#rTrRrrrrrrrrrAps (  -  rAc@seZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ e e e Z ddZe eZeddZddZddZddZdS) EncodingByteszString-like object with an associated position and various extra methods If the position is ever greater than the string length then an exception is raisedcCs+t|tsttj||jS)N)r>r/r)__new__r)r"valuerrrr&szEncodingBytes.__new__cCs d|_dS)Nrr)rb)r"rrrrr#*szEncodingBytes.__init__cCs|S)Nr)r"rrr__iter__-szEncodingBytes.__iter__cCsV|jd}|_|t|kr/tn|dkrDtn|||dS)Nrr)rbr$ StopIterationr?)r"prrr__next__0s    zEncodingBytes.__next__cCs |jS)N)r)r"rrrnext8szEncodingBytes.nextcCs\|j}|t|kr$tn|dkr9tn|d|_}|||dS)Nrr)rbr$rr?)r"rrrrprevious<s    zEncodingBytes.previouscCs+|jt|krtn||_dS)N)rbr$r)r"r!rrr setPositionEs zEncodingBytes.setPositioncCs<|jt|krtn|jdkr4|jSdSdS)Nr)rbr$r)r"rrr getPositionJs  zEncodingBytes.getPositioncCs||j|jdS)Nr)r!)r"rrrgetCurrentByteTszEncodingBytes.getCurrentBytecCsf|j}xM|t|krX|||d}||krK||_|S|d7}q W||_dS)zSkip past a list of charactersrN)r!r$rb)r"rrr}rrrrxYs    zEncodingBytes.skipcCsf|j}xM|t|krX|||d}||krK||_|S|d7}q W||_dS)Nr)r!r$rb)r"rrr}rrr skipUntiles    zEncodingBytes.skipUntilcCsT|j}|||t|}|j|}|rP|jt|7_n|S)zLook for a sequence of bytes at the start of a string. If the bytes are found return True and advance the position to the byte after the match. Otherwise return False and leave the position alone)r!r$ startswith)r"r/rr3r7rrr matchBytesps  zEncodingBytes.matchBytescCsn||jdj|}|dkrd|jdkrCd|_n|j|t|d7_dStdS)zLook for the next sequence of bytes matching a given sequence. If a match is found advance the position to the last byte of the matchNrrTrr)r!findrbr$r)r"r/ newPositionrrrjumpTo{s  zEncodingBytes.jumpToN)rrrr<rr#rrrrrrpropertyr!r currentBytespaceCharactersBytesrxrrrrrrrr"s           rc@seZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ dS)rz?Mini parser for detecting character encoding from meta elementscCst||_d|_dS)z3string - the data to work on for encoding detectionN)rr3rC)r"r3rrrr#szEncodingParser.__init__c Csd|jfd|jfd|jfd|jfd|jfd|jff}xw|jD]l}d}xS|D]K\}}|jj|rky|}PWqtk rd}PYqXqkqkW|sXPqXqXW|jS) Ns)r3r)r"rrrrszEncodingParser.handleCommentcCsE|jjtkrdSd}d}x|j}|dkrAdS|ddkr|ddk}|r=|dk r=||_dSq%|ddkr|d}t|}|dk r=||_dSq%|ddkr%tt|d}|j}|dk r=t|}|dk r:|r.||_dS|}q:q=q%q%WdS) NTFrs http-equivrs content-typescharsetscontent) r3rr getAttributerCrContentAttrParserrparse)r" hasPragmapendingEncodingattrtentativeEncodingcodec contentParserrrrrs:            zEncodingParser.handleMetacCs |jdS)NF)handlePossibleTag)r"rrrrsz%EncodingParser.handlePossibleStartTagcCst|j|jdS)NT)rr3r)r"rrrrs z#EncodingParser.handlePossibleEndTagcCs|j}|jtkr9|r5|j|jndS|jt}|dkra|jn+|j}x|dk r|j}qpWdS)NTr)r3rasciiLettersBytesrrrspacesAngleBracketsr)r"endTagr3r}rrrrrs      z EncodingParser.handlePossibleTagcCs|jjdS)Nr)r3r)r"rrrrszEncodingParser.handleOthercCs|j}|jttdgB}|dksIt|dksIt|d krYdSg}g}x|dkr~|r~Pnz|tkr|j}Pn^|d krdj|dfS|tkr|j|j n|dkrdS|j|t |}qhW|dkr1|j dj|dfSt ||j}|d kr|}xt |}||krt |dj|dj|fS|tkr|j|j q\|j|q\Wn^|dkrdj|dfS|tkr|j|j n|dkr!dS|j|xwt |}|t kredj|dj|fS|tkr|j|j q1|dkrdS|j|q1WdS) z_Return a name,value pair for the next attribute in the stream, if one is found, or None/Nrr='")rN)rr)rr) r3rxr frozensetr$r)r5asciiUppercaseBytesr2rrrr)r"r3r}attrName attrValue quoteCharrrrrsh $                        zEncodingParser.getAttributeN) rrrr<r#rrrrrrrrrrrrrs     $    rc@s(eZdZddZddZdS)rcCs"t|tst||_dS)N)r>r/r)r3)r"r3rrrr#?szContentAttrParser.__init__cCsNy1|jjd|jjd7_|jj|jjdksHdS|jjd7_|jj|jjdkr|jj}|jjd7_|jj}|jj|r|j||jjSdSn]|jj}y+|jjt|j||jjSWn#tk r/|j|dSYnXWntk rIdSYnXdS)Nscharsetrrrr)rr)r3rr!rxrrrr)r" quoteMark oldPositionrrrrCs.       zContentAttrParser.parseN)rrrr#rrrrrr>s  rc Cstt|tr>y|jd}Wq>tk r:dSYq>Xn|rltjd|j}tj|dSdSdS)z{Return the python codec name corresponding to an encoding or None if the string doesn't correspond to a valid encoding.rNr4) r>r/decodeUnicodeDecodeErrorascii_punctuation_rerkrr r)rC canonicalNamerrrres  r)+ __future__rrrZpip._vendor.sixrrrL constantsrrr r r r r4r iorrrrobjectrrrrrrMrosetrwrr~rrGr@rAr/rrrrrrrrsJ  "   Jg'