U Tr­`peã@sddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZddlZddlmZddlmZddlmZdd gZdZd Zd Zd  eee¡ZGd d „d eƒZ Gdd„deƒZ!Gdd„dƒZ"dd„Z#dS)éN)ÚStringIO)Úc_inté)Úutils)Ú ControlThread)ÚdownloadÚSmartDLréz{}.{}.{}c@s(eZdZdZdd„Zdd„Zdd„ZdS) ÚHashFailedExceptionzRaised when hash check fails.cCs||_||_||_dS©N)ÚfilenameÚcalculated_hashÚ needed_hash)ÚselfÚfnZ calc_hashr©rúJ/mnt/cestravail/FreezeDetection/sentSatProg_v.0.1.3/pySmartDL/pySmartDL.pyÚ__init__szHashFailedException.__init__cCsd |j|j|j¡S)Nz,HashFailedException({}, got {}, expected {})©Úformatr r r©rrrrÚ__str__szHashFailedException.__str__cCsd |j|j|j¡S)Nz-rrrrrÚ__repr__!szHashFailedException.__repr__N©Ú__name__Ú __module__Ú __qualname__Ú__doc__rrrrrrrr sr c@s(eZdZdZdd„Zdd„Zdd„ZdS) ÚCanceledExceptionz Raised when the job is canceled.cCsdSr rrrrrr&szCanceledException.__init__cCsdS)Nrrrrrrr(szCanceledException.__str__cCsdS)Nzrrrrrr*szCanceledException.__repr__Nrrrrrr$src @s eZdZdZdIdd„Zd d „Zd d „Zd d„Zdd„Zdd„Z dJdd„Z dd„Z dKdd„Z dLdd„Z dMdd„ZdNdd„Zd d!„ZdOd$d%„Zd&d'„Zd(d)„Zd*d+„Zd,d-„ZdPd.d/„Zd0d1„Zd2d3„Zd4d5„Zd6d7„Zd8d9„Zd:d;„ZdQdd?„ZdSd@dA„ZdTdCdD„Z dEdF„Z!dGdH„Z"dS)Uram The main SmartDL class :param urls: Download url. It is possible to pass unsafe and unicode characters. You can also pass a list of urls, and those will be used as mirrors. :type urls: string or list of strings :param dest: Destination path. Default is `%TEMP%/pySmartDL/`. :type dest: string :param progress_bar: If True, prints a progress bar to the `stdout stream `_. Default is `True`. :type progress_bar: bool :param fix_urls: If true, attempts to fix urls with unsafe characters. :type fix_urls: bool :param threads: Number of threads to use. :type threads: int :param timeout: Timeout for network operations, in seconds. Default is 5. :type timeout: int :param logger: An optional logger. :type logger: `logging.Logger` instance :param connect_default_logger: If true, connects a default logger to the class. :type connect_default_logger: bool :param request_args: Arguments to be passed to a new urllib.request.Request instance in dictionary form. See `urllib.request docs `_ for options. :type request_args: dict :rtype: `SmartDL` instance .. NOTE:: The provided dest may be a folder or a full path name (including filename). The workflow is: * If the path exists, and it's an existing folder, the file will be downloaded to there with the original filename. * If the past does not exist, it will create the folders, if needed, and refer to the last section of the path as the filename. * If you want to download to folder that does not exist at the moment, and want the module to fill in the filename, make sure the path ends with `os.sep`. * If no path is provided, `%TEMP%/pySmartDL/` will be used. NTééFcCs|r ||_n|rt ¡|_n t ¡|_| rDd| krcsz$SmartDL.__init__..rúUsing url "{}"Ú pySmartDLéÿÿÿÿréi ÚreadyFTz'Folder "{}" does not exist. Creating...)r!Útimeoutz=Server does not support HTTPRange. threads_count is set to 1.z?Destination "{}" already exists. Existing file will be removed.z-Directory "{}" does not exist. Creating it...z&Creating a ThreadPool of {} thread(s).):ÚloggerrZcreate_debugging_loggerZ DummyLoggerÚdictÚ requestArgsÚadd_basic_authenticationÚ isinstanceÚstrÚmirrorsÚpopÚurlÚinforÚurllibÚparseÚunquoteÚosÚpathÚbasenameÚurlparseÚjoinÚtempfileÚ gettempdirÚdestÚsepÚexistsÚisfileÚunlinkÚisdirÚ progress_barÚ threads_countr+Úcurrent_attempÚ attemps_limitÚ minChunkFileÚfilesizeÚmultiprocessingÚValuerÚ shared_varÚthread_shared_cmdsÚstatusÚ verify_hashÚ_killedÚ_failedÚ_start_func_blockingÚerrorsÚhash_algorithmÚ hash_codeÚpost_threadpool_threadÚcontrol_threadÚdirnameÚmakedirsZauto_detect_HTTPRange_supportÚwarningZManagedThreadPoolExecutorÚpool)rZurlsr@rFZfix_urlsÚthreadsr+r,Zconnect_default_loggerZ request_argsZ hash_checksumZ hash_algoÚusernameÚpasswordrrrrrNsv      0 zSmartDL.__init__cCsd |j|j¡S)NzSmartDL(r"{}", dest=r"{}"))rr4r@rrrrr”szSmartDL.__str__cCs d |j¡S)Nz )rr4rrrrr—szSmartDL.__repr__cCs2d ||¡}t | d¡¡}d||jdd<dS)zØ Uses HTTP Basic Access authentication for the connection. :param username: Username. :type username: string :param password: Password. :type password: string z{}:{}zutf-8sBasic r!Ú AuthorizationN)rÚbase64Ústandard_b64encodeÚencoder.)rr_r`Z auth_stringZ base64stringrrrr/šs z SmartDL.add_basic_authenticationcCsd|_||_||_dS)aÒ Adds hash verification to the download. If hash is not correct, will try different mirrors. If all mirrors aren't passing hash verification, `HashFailedException` Exception will be raised. .. NOTE:: If downloaded file already exist on the destination, and hash matches, pySmartDL will not download it again. .. WARNING:: The hashing algorithm must be supported on your system, as documented at `hashlib documentation page `_. :param algorithm: Hashing algorithm. :type algorithm: string :param hash: Hash code. :type hash: string TN)rQrVrW)rÚ algorithmÚhashrrrÚadd_hash_verification§szSmartDL.add_hash_verificationc Csúdddg}tj |j¡}tj |j¡}|j d¡|D]¾}zœd||f}tjj |f|j Ž}tj  |¡}|  ¡  d¡}| ¡|D]R} | ¡|  ¡kr€|j d|¡| d¡} |   d ¡d } | | | ¡Wd Sq€Wq6tjjk ròYq6Yq6Xq6d S) a4 Will attempt to fetch UNIX hash sums files (`SHA256SUMS`, `SHA1SUMS` or `MD5SUMS` files in the same url directory). Calls `self.add_hash_verification` if successful. Returns if a matching hash was found. :rtype: bool *New in 1.2.1* Z SHA256SUMSZSHA1SUMSZMD5SUMSzLooking for SUMS files...z%s/%sÚ zFound a matching hash in %sZSUMSú rN)r9r:rZr4r;r,r5r6ÚrequestÚRequestr.ÚurlopenÚreadÚsplitÚcloseÚlowerÚrstriprgÚerrorÚ HTTPError) rZdefault_sums_filenamesÚfolderZ orig_basenamer Zsums_urlZ sumsRequestÚobjÚdataÚlineZalgorfrrrÚfetch_hash_sums¾s(      zSmartDL.fetch_hash_sumsc sXˆjdkstd ˆj¡ƒ‚ˆj d¡|dkr6ˆj}n|ˆ_ˆjr\ˆj d tˆjƒ¡¡n ˆj d¡ˆjdk rºt j   ˆj ¡rºt  ˆjˆj ¡ ¡ˆj ¡krºˆj dˆj ¡dˆ_dSˆj d  ˆjˆj ¡¡tjjˆjfˆjŽ}ztjj|ˆjd }Wn¾tjjtjjtjfk rº}zŽˆj |¡ˆjr€ˆj d  t|ƒ¡¡ˆj d ¡ˆ_ˆj d  ˆj¡¡ˆ  |¡WY¢2dSˆj !t|ƒ¡ˆj |¡dˆ_"dˆ_‚W5d}~XYnXz2t#|j$dƒˆ_%ˆj d ˆj%t  &ˆj%¡¡¡Wn.t't(t)fk rˆj !d¡d ˆ_%YnXt  *ˆj%ˆj+ˆj,¡}|d d|d d d}t|ƒdkr|ˆj d t|ƒt  &|¡¡¡nˆj d t  &|¡¡¡dˆ_t-|ƒD]J\}}ˆj. /t0ˆjˆj d|ˆj|d |dˆjdˆj1ˆj2ˆjd¡ }q¢t3j4t5ˆj.‡fdd„t6t|ƒƒDƒˆj gˆj%ˆfdˆ_7dˆj7_8ˆj7  ¡t9ˆƒˆ_:|rTˆj;dddS)a, Starts the download task. Will raise `RuntimeError` if it's the object's already downloading. .. warning:: If you're using the non-blocking mode, Exceptions won't be raised. In that case, call `isSuccessful()` after the task is finished, to make sure the download succeeded. Call `get_errors()` to get the the exceptions. :param blocking: If true, calling this function will block the thread until the download finished. Default is *True*. :type blocking: bool r*z#cannot start (current status is {})z!Starting a new SmartDL operation.Nz"One URL and {} mirrors are loaded.zOne URL is loaded.zKDestination '%s' already exists, and the hash matches. No need to download.ÚfinishedzDownloading '{}' to '{}'...)r+z{} Trying next mirror...rr&TzContent-LengthzContent-Length is {} ({}).z8Server did not send Content-Length. Filesize is unknown.rz+Launching {} threads (downloads {}/thread).z"Launching 1 thread (downloads {}).Ú downloadingú.%.3dFcsg|]}ˆjd|‘qS)r{©r@)r#Úirrrr%8sz!SmartDL.start..)ÚtargetÚargs)Úraise_exceptions)||j|jd<nd|jkrP|jd=dS)zÐ Limits the download transfer speed. :param speed: Speed in bytes per download per second. Negative values will not limit the speed. Default is `-1`. :type speed: int rzrÚlimitN)rPrºr¼rGrO)rÚspeedrrrÚ limit_speeds   zSmartDL.limit_speedcCs|jS)z¼ Get the destination path of the downloaded file. Needed when no destination is provided to the class, and exists on a temp folder. :rtype: string r|rrrrÚget_destszSmartDL.get_destcCs(|js dS|rt |j ¡¡S|j ¡S)a* Returns how much time did the download take, in seconds. Returns `-1` if the download task is not finished yet. :param human: If true, returns a human-readable formatted string. Else, returns an int type number :type human: bool :rtype: int/string r)rYrr¡Ú get_dl_timer¥rrrrÂs zSmartDL.get_dl_timecCs(|js dS|rt |j ¡¡S|j ¡S)zá Get downloaded bytes counter in bytes. :param human: If true, returns a human-readable formatted string. Else, returns an int type number :type human: bool :rtype: int/string r)rYrr‰r¦r¥rrrr¦+s zSmartDL.get_dl_sizecCs(|js dS|rt |j ¡¡S|j ¡S)zÜ Get total download size in bytes. :param human: If true, returns a human-readable formatted string. Else, returns an int type number :type human: bool :rtype: int/string r)rYrr‰Úget_final_filesizer¥rrrrÃ9s zSmartDL.get_final_filesizer(c Cs\|jdkrtd|jƒ‚|r dnd}t| ¡|ƒ }|dkrF| |¡n| ¡}W5QRX|S)až Returns the downloaded data. Will raise `RuntimeError` if it's called when the download task is not finished yet. :param binary: If true, will read the data as binary. Else, will read it as text. :type binary: bool :param bytes: Number of bytes to read. Negative values will read until EOF. Default is `-1`. :type bytes: int :rtype: string ryzTThe download task must be finished in order to read the data. (current status is %s)ÚrbÚrr)rPrÚopenrÁrm)rÚbinaryÚbytesÚflagsÚfrvrrrÚget_dataHs  $zSmartDL.get_datacCst ||jdd¡ ¡S)a¿ Returns the downloaded data's hash. Will raise `RuntimeError` if it's called when the download task is not finished yet. :param algorithm: Hashing algorithm. :type algorithm: bool :rtype: string .. WARNING:: The hashing algorithm must be supported on your system, as documented at `hashlib documentation page `_. T)rÇ)ÚhashlibÚnewrËÚ hexdigest)rrerrrÚ get_data_hash[s zSmartDL.get_data_hashcCs| ¡}t |¡S)a Returns the JSON in the downloaded data. Will raise `RuntimeError` if it's called when the download task is not finished yet. Will raise `json.decoder.JSONDecodeError` if the downloaded data is not valid JSON. :rtype: dict )rËÚjsonÚloads)rrvrrrÚget_jsoniszSmartDL.get_json) NTTrr NFNNNNN)N)r›)N)F)F)r¨r«)F)F)F)F)Fr()#rrrrrrrr/rgrxr‡ršrŸr r¢r¤r§r­r¯rµr¶r·r”r¸rºr½r¼rÀrÁrÂr¦rÃrËrÏrÒrrrrr-s@ F # d          c CsR| ¡st d¡q|jrdS| ¡rR| ¡D]}|j |¡q.| t | ¡ƒ¡|j rh|j  d¡dS|rÒt |dƒ}t dd„|dDƒƒ}t ||¡}|d|krÒd ||||¡}|j  |¡| |¡dSd |_|j d ¡tj|Ž|jrN|d } t |j| ¡} | |jkr&|j d ¡n(|j  d ¡| ttj | ¡t|jƒ¡dS)z=Run function after thread pool is done. Run this in a thread.r°NzTask had errors. Exiting...rcSsg|]}tj |¡‘qSr)r9r:Úgetsizer"rrrr%ˆsz+post_threadpool_actions..iznDiff between downloaded files and expected filesizes is {}B (filesize: {}, expected_filesize: {}, {} threads).Ú combiningzCombining downloaded file.r(zHash verification succeeded.zHash verification failed.)Údoner²r³rRZ get_exceptionZget_exceptionsr,r™rŸr1rSr\r‚ÚsumÚmathÚfabsrrPr5rZ combine_filesrQrƒrVrWr r r9r:r;rf) r]rZexpected_filesizeZ SmartDLObjÚexcr^Ztotal_filesizeÚdiffZerrMsgZ dest_pathZhash_rrrr‘ts<           r‘)$r9Úurllib.requestr6Ú urllib.errorÚ urllib.parserr²r×r>rbrÌr…ÚiorZmultiprocessing.dummyÚdummyrLÚctypesrrÐr›rrYrrÚ__all__Z__version_mjaor__Z__version_minor__Z__version_micro__rÚ __version__Ú Exceptionr rrr‘rrrrÚs8        K