B ÛÞß]peã@sddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZddlZddlmZddlmZddlmZdd gZdZd Zd Zd  eee¡ZGd d „d eƒZ Gdd„deƒZ!Gdd„dƒZ"dd„Z#dS)éN)ÚStringIO)Úc_inté)Úutils)Ú ControlThread)ÚdownloadÚSmartDLréz{}.{}.{}c@s(eZdZdZdd„Zdd„Zdd„ZdS) ÚHashFailedExceptionzRaised when hash check fails.cCs||_||_||_dS)N)ÚfilenameÚcalculated_hashÚ needed_hash)ÚselfÚfnZ calc_hashr ©rúIC:\Users\ibrahim.fayad\Desktop\sentSatProg_v.0.1.3\pySmartDL\pySmartDL.pyÚ__init__szHashFailedException.__init__cCsd |j|j|j¡S)Nz,HashFailedException({}, got {}, expected {}))Úformatr r r )rrrrÚ__str__szHashFailedException.__str__cCsd |j|j|j¡S)Nz-)rr r r )rrrrÚ__repr__!szHashFailedException.__repr__N)Ú__name__Ú __module__Ú __qualname__Ú__doc__rrrrrrrr sr c@s(eZdZdZdd„Zdd„Zdd„ZdS) ÚCanceledExceptionz Raised when the job is canceled.cCsdS)Nr)rrrrr&szCanceledException.__init__cCsdS)Nrr)rrrrr(szCanceledException.__str__cCsdS)Nzr)rrrrr*szCanceledException.__repr__N)rrrrrrrrrrrr$src @s eZdZdZdIdd„Zd d „Zd d „Zd d„Zdd„Zdd„Z dJdd„Z dd„Z dKdd„Z dLdd„Z dMdd„ZdNdd„Zd d!„ZdOd$d%„Zd&d'„Zd(d)„Zd*d+„Zd,d-„ZdPd.d/„Zd0d1„Zd2d3„Zd4d5„Zd6d7„Zd8d9„Zd:d;„ZdQdd?„ZdSd@dA„ZdTdCdD„Z dEdF„Z!dGdH„Z"dS)Uram The main SmartDL class :param urls: Download url. It is possible to pass unsafe and unicode characters. You can also pass a list of urls, and those will be used as mirrors. :type urls: string or list of strings :param dest: Destination path. Default is `%TEMP%/pySmartDL/`. :type dest: string :param progress_bar: If True, prints a progress bar to the `stdout stream `_. Default is `True`. :type progress_bar: bool :param fix_urls: If true, attempts to fix urls with unsafe characters. :type fix_urls: bool :param threads: Number of threads to use. :type threads: int :param timeout: Timeout for network operations, in seconds. Default is 5. :type timeout: int :param logger: An optional logger. :type logger: `logging.Logger` instance :param connect_default_logger: If true, connects a default logger to the class. :type connect_default_logger: bool :param request_args: Arguments to be passed to a new urllib.request.Request instance in dictionary form. See `urllib.request docs `_ for options. :type request_args: dict :rtype: `SmartDL` instance .. NOTE:: The provided dest may be a folder or a full path name (including filename). The workflow is: * If the path exists, and it's an existing folder, the file will be downloaded to there with the original filename. * If the past does not exist, it will create the folders, if needed, and refer to the last section of the path as the filename. * If you want to download to folder that does not exist at the moment, and want the module to fill in the filename, make sure the path ends with `os.sep`. * If no path is provided, `%TEMP%/pySmartDL/` will be used. NTééFcCs|r ||_n|rt ¡|_n t ¡|_| rDd| krcsz$SmartDL.__init__..rzUsing url "{}"Ú pySmartDLéÿÿÿÿréi ÚreadyFTz'Folder "{}" does not exist. Creating...)rÚtimeoutz=Server does not support HTTPRange. threads_count is set to 1.z?Destination "{}" already exists. Existing file will be removed.z-Directory "{}" does not exist. Creating it...z&Creating a ThreadPool of {} thread(s).):ÚloggerrZcreate_debugging_loggerZ DummyLoggerÚdictÚ requestArgsÚadd_basic_authenticationÚ isinstanceÚstrÚmirrorsÚpopÚurlÚinforÚurllibÚparseÚunquoteÚosÚpathÚbasenameÚurlparseÚjoinÚtempfileÚ gettempdirÚdestÚsepÚexistsÚisfileÚunlinkÚisdirÚ progress_barÚ threads_countr%Úcurrent_attempÚ attemps_limitÚ minChunkFileÚfilesizeÚmultiprocessingÚValuerÚ shared_varÚthread_shared_cmdsÚstatusÚ verify_hashÚ_killedÚ_failedÚ_start_func_blockingÚerrorsÚhash_algorithmÚ hash_codeÚpost_threadpool_threadÚcontrol_threadÚdirnameÚmakedirsZauto_detect_HTTPRange_supportÚwarningZManagedThreadPoolExecutorÚpool)rZurlsr:r@Zfix_urlsÚthreadsr%r&Zconnect_default_loggerZ request_argsZ hash_checksumZ hash_algoÚusernameÚpasswordrrrrrNsv      0 zSmartDL.__init__cCsd |j|j¡S)NzSmartDL(r"{}", dest=r"{}"))rr.r:)rrrrr”szSmartDL.__str__cCs d |j¡S)Nz )rr.)rrrrr—szSmartDL.__repr__cCs2d ||¡}t | d¡¡}d||jdd<dS)zØ Uses HTTP Basic Access authentication for the connection. :param username: Username. :type username: string :param password: Password. :type password: string z{}:{}zutf-8sBasic rÚ AuthorizationN)rÚbase64Ústandard_b64encodeÚencoder()rrYrZZ auth_stringZ base64stringrrrr)šs z SmartDL.add_basic_authenticationcCsd|_||_||_dS)aÒ Adds hash verification to the download. If hash is not correct, will try different mirrors. If all mirrors aren't passing hash verification, `HashFailedException` Exception will be raised. .. NOTE:: If downloaded file already exist on the destination, and hash matches, pySmartDL will not download it again. .. WARNING:: The hashing algorithm must be supported on your system, as documented at `hashlib documentation page `_. :param algorithm: Hashing algorithm. :type algorithm: string :param hash: Hash code. :type hash: string TN)rKrPrQ)rÚ algorithmÚhashrrrÚadd_hash_verification§szSmartDL.add_hash_verificationc Csúdddg}tj |j¡}tj |j¡}|j d¡xÂ|D]º}yšd||f}tjj |f|j Ž}tj  |¡}|  ¡  d¡}| ¡xT|D]L} | ¡|  ¡kr„|j d|¡| d¡} |   d ¡d } | | | ¡d Sq„WWq8tjjk rðw8Yq8Xq8Wd S) a4 Will attempt to fetch UNIX hash sums files (`SHA256SUMS`, `SHA1SUMS` or `MD5SUMS` files in the same url directory). Calls `self.add_hash_verification` if successful. Returns if a matching hash was found. :rtype: bool *New in 1.2.1* Z SHA256SUMSZSHA1SUMSZMD5SUMSzLooking for SUMS files...z%s/%sÚ zFound a matching hash in %sZSUMSú rN)r3r4rTr.r5r&r/r0ÚrequestÚRequestr(ÚurlopenÚreadÚsplitÚcloseÚlowerÚrstripraÚerrorÚ HTTPError) rZdefault_sums_filenamesZfolderZ orig_basenamer Zsums_urlZ sumsRequestÚobjÚdataÚlineZalgor`rrrÚfetch_hash_sums¾s(         zSmartDL.fetch_hash_sumsc sVˆjdkstd ˆj¡ƒ‚ˆj d¡|dkr6ˆj}n|ˆ_ˆjr\ˆj d tˆjƒ¡¡n ˆj d¡ˆjdk rºt j   ˆj ¡rºt  ˆjˆj ¡ ¡ˆj ¡krºˆj dˆj ¡dˆ_dSˆj d  ˆjˆj ¡¡tjjˆjfˆjŽ}ytjj|ˆjd }Wn¸tjjtjjtjfk r´}zˆˆj |¡ˆjrzˆj d  t|ƒ¡¡ˆj d ¡ˆ_ˆj d  ˆj¡¡ˆ  |¡dSˆj !t|ƒ¡ˆj |¡dˆ_"dˆ_‚Wdd}~XYnXy2t#|j$dƒˆ_%ˆj d ˆj%t  &ˆj%¡¡¡Wn.t't(t)fk rˆj !d¡d ˆ_%YnXt  *ˆj%ˆj+ˆj,¡}|d d|d d d}t|ƒdkrvˆj d t|ƒt  &|¡¡¡nˆj d t  &|¡¡¡dˆ_xVt-|ƒD]J\}}ˆj. /t0ˆjˆj d|ˆj|d |dˆjdˆj1ˆj2ˆjd¡ }qžWt3j4t5ˆj.‡fdd„t6t|ƒƒDƒˆj gˆj%ˆfdˆ_7dˆj7_8ˆj7  ¡t9ˆƒˆ_:|rRˆj;dddS)a, Starts the download task. Will raise `RuntimeError` if it's the object's already downloading. .. warning:: If you're using the non-blocking mode, Exceptions won't be raised. In that case, call `isSuccessful()` after the task is finished, to make sure the download succeeded. Call `get_errors()` to get the the exceptions. :param blocking: If true, calling this function will block the thread until the download finished. Default is *True*. :type blocking: bool r$z#cannot start (current status is {})z!Starting a new SmartDL operation.Nz"One URL and {} mirrors are loaded.zOne URL is loaded.zKDestination '%s' already exists, and the hash matches. No need to download.ÚfinishedzDownloading '{}' to '{}'...)r%z{} Trying next mirror...rzUsing url "{}"TzContent-LengthzContent-Length is {} ({}).z8Server did not send Content-Length. Filesize is unknown.rz+Launching {} threads (downloads {}/thread).z"Launching 1 thread (downloads {}).Ú downloadingz.%.3dFcsg|]}ˆjd|‘qS)z.%.3d)r:)rÚi)rrrr 8sz!SmartDL.start..)ÚtargetÚargs)Úraise_exceptions)||j|jd<nd|jkrP|jd=dS)zÐ Limits the download transfer speed. :param speed: Speed in bytes per download per second. Negative values will not limit the speed. Default is `-1`. :type speed: int rsrÚlimitN)rJr®r¯rArI)rZspeedrrrÚ limit_speeds   zSmartDL.limit_speedcCs|jS)z¼ Get the destination path of the downloaded file. Needed when no destination is provided to the class, and exists on a temp folder. :rtype: string )r:)rrrrÚget_destszSmartDL.get_destcCs(|js dS|rt |j ¡¡S|j ¡S)a* Returns how much time did the download take, in seconds. Returns `-1` if the download task is not finished yet. :param human: If true, returns a human-readable formatted string. Else, returns an int type number :type human: bool :rtype: int/string r)rSrr—Ú get_dl_time)rr™rrrr´s zSmartDL.get_dl_timecCs(|js dS|rt |j ¡¡S|j ¡S)zá Get downloaded bytes counter in bytes. :param human: If true, returns a human-readable formatted string. Else, returns an int type number :type human: bool :rtype: int/string r)rSrr€r›)rr™rrrr›+s zSmartDL.get_dl_sizecCs(|js dS|rt |j ¡¡S|j ¡S)zÜ Get total download size in bytes. :param human: If true, returns a human-readable formatted string. Else, returns an int type number :type human: bool :rtype: int/string r)rSrr€Úget_final_filesize)rr™rrrrµ9s zSmartDL.get_final_filesizer"c Cs\|jdkrtd|jƒ‚|r dnd}t| ¡|ƒ }|dkrF| |¡n| ¡}WdQRX|S)až Returns the downloaded data. Will raise `RuntimeError` if it's called when the download task is not finished yet. :param binary: If true, will read the data as binary. Else, will read it as text. :type binary: bool :param bytes: Number of bytes to read. Negative values will read until EOF. Default is `-1`. :type bytes: int :rtype: string rrzTThe download task must be finished in order to read the data. (current status is %s)ÚrbÚrrN)rJrxÚopenr³rg)rÚbinaryÚbytesÚflagsÚfrorrrÚget_dataHs  $zSmartDL.get_datacCst ||jdd¡ ¡S)a¿ Returns the downloaded data's hash. Will raise `RuntimeError` if it's called when the download task is not finished yet. :param algorithm: Hashing algorithm. :type algorithm: bool :rtype: string .. WARNING:: The hashing algorithm must be supported on your system, as documented at `hashlib documentation page `_. T)r¹)ÚhashlibÚnewr½Ú hexdigest)rr_rrrÚ get_data_hash[s zSmartDL.get_data_hashcCs| ¡}t |¡S)a Returns the JSON in the downloaded data. Will raise `RuntimeError` if it's called when the download task is not finished yet. Will raise `json.decoder.JSONDecodeError` if the downloaded data is not valid JSON. :rtype: dict )r½ÚjsonÚloads)rrorrrÚget_jsoniszSmartDL.get_json) NTTrrNFNNNNN)N)r‘)N)F)F)rr )F)F)F)F)Fr")#rrrrrrrr)rarqr~rr•r–r˜ršrœr¢r¤r©rªr«rŠr¬r®r°r¯r²r³r´r›rµr½rÁrÄrrrrr-s@ F # d          c CsZx| ¡st d¡qW|jr"dS| ¡rZx| ¡D]}|j |¡q4W| t | ¡ƒ¡|j rp|j  d¡dS|rÚt |dƒ}t dd„|dDƒƒ}t ||¡}|d|krÚd ||||¡}|j  |¡| |¡dSd |_|j d ¡tj|Ž|jrV|d } t |j| ¡} | |jkr.|j d ¡n(|j  d ¡| ttj | ¡t|jƒ¡dS)z=Run function after thread pool is done. Run this in a thread.gš™™™™™¹?NzTask had errors. Exiting...rcSsg|]}tj |¡‘qSr)r3r4Úgetsize)rrrrrr ˆsz+post_threadpool_actions..iznDiff between downloaded files and expected filesizes is {}B (filesize: {}, expected_filesize: {}, {} threads).Ú combiningzCombining downloaded file.r"zHash verification succeeded.zHash verification failed.)Údoner¦r§rLZ get_exceptionZget_exceptionsr&rr•r+rMrVryÚsumÚmathÚfabsrrJr/rZ combine_filesrKrzrPrQr–r r3r4r5r`) rWrvZexpected_filesizeZ SmartDLObjÚexcrXZtotal_filesizeZdiffZerrMsgZ dest_pathZhash_rrrr‡ts<          r‡)$r3Úurllib.requestr0Ú urllib.errorÚ urllib.parser…r¦rÉr8r\r¾r|ÚiorZmultiprocessing.dummyÚdummyrFÚctypesrrÂr‘rrSrrÚ__all__Z__version_mjaor__Z__version_minor__Z__version_micro__rÚ __version__Ú Exceptionr rrr‡rrrrÚs8        K