3 ]pe@sddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z ddljZddlmZddlZddlmZddlmZddlmZdd gZdZd Zd Zd jeeeZGd d d eZ GdddeZ!GdddZ"ddZ#dS)N)StringIO)c_int)utils) ControlThread)downloadSmartDLrz{}.{}.{}c@s(eZdZdZddZddZddZdS) HashFailedExceptionzRaised when hash check fails.cCs||_||_||_dS)N)filenamecalculated_hash needed_hash)selffnZ calc_hashr rD/home/ghaith/Applications/sentSatProg_v.0.1.3/pySmartDL/pySmartDL.py__init__szHashFailedException.__init__cCsdj|j|j|jS)Nz,HashFailedException({}, got {}, expected {}))formatr r r )rrrr__str__szHashFailedException.__str__cCsdj|j|j|jS)Nz-)rr r r )rrrr__repr__!szHashFailedException.__repr__N)__name__ __module__ __qualname____doc__rrrrrrrr sr c@s(eZdZdZddZddZddZdS) CanceledExceptionz Raised when the job is canceled.cCsdS)Nr)rrrrr&szCanceledException.__init__cCsdS)Nrr)rrrrr(szCanceledException.__str__cCsdS)Nzr)rrrrr*szCanceledException.__repr__N)rrrrrrrrrrrr$src @s eZdZdZdJddZd d Zd d Zd dZddZddZ dKddZ ddZ dLddZ dMddZ dNddZdOddZd d!ZdQd%d&Zd'd(Zd)d*Zd+d,Zd-d.ZdRd/d0Zd1d2Zd3d4Zd5d6Zd7d8Zd9d:Zd;d<ZdSd=d>ZdTd?d@ZdUdAdBZdWdDdEZ dFdGZ!dHdIZ"dS)Xram The main SmartDL class :param urls: Download url. It is possible to pass unsafe and unicode characters. You can also pass a list of urls, and those will be used as mirrors. :type urls: string or list of strings :param dest: Destination path. Default is `%TEMP%/pySmartDL/`. :type dest: string :param progress_bar: If True, prints a progress bar to the `stdout stream `_. Default is `True`. :type progress_bar: bool :param fix_urls: If true, attempts to fix urls with unsafe characters. :type fix_urls: bool :param threads: Number of threads to use. :type threads: int :param timeout: Timeout for network operations, in seconds. Default is 5. :type timeout: int :param logger: An optional logger. :type logger: `logging.Logger` instance :param connect_default_logger: If true, connects a default logger to the class. :type connect_default_logger: bool :param request_args: Arguments to be passed to a new urllib.request.Request instance in dictionary form. See `urllib.request docs `_ for options. :type request_args: dict :rtype: `SmartDL` instance .. NOTE:: The provided dest may be a folder or a full path name (including filename). The workflow is: * If the path exists, and it's an existing folder, the file will be downloaded to there with the original filename. * If the past does not exist, it will create the folders, if needed, and refer to the last section of the path as the filename. * If you want to download to folder that does not exist at the moment, and want the module to fill in the filename, make sure the path ends with `os.sep`. * If no path is provided, `%TEMP%/pySmartDL/` will be used. NTFcCs|r ||_n|rtj|_n tj|_| rDd| krcsz$SmartDL.__init__..rzUsing url "{}" pySmartDLrirreadyFTz'Folder "{}" does not exist. Creating...)rtimeoutz=Server does not support HTTPRange. threads_count is set to 1.z?Destination "{}" already exists. Existing file will be removed.z-Directory "{}" does not exist. Creating it...z&Creating a ThreadPool of {} thread(s).r%r%r%ii ):loggerrZcreate_debugging_loggerZ DummyLoggerdict requestArgsadd_basic_authentication isinstancestrmirrorspopurlinforurllibparseunquoteospathbasenameurlparsejointempfile gettempdirdestsepexistsisfileunlinkisdir progress_bar threads_countr$current_attemp attemps_limit minChunkFilefilesizemultiprocessingValuer shared_varthread_shared_cmdsstatus verify_hash_killed_failed_start_func_blockingerrorshash_algorithm hash_codepost_threadpool_threadcontrol_threaddirnamemakedirsZauto_detect_HTTPRange_supportwarningZManagedThreadPoolExecutorpool)rZurlsr:r@Zfix_urlsthreadsr$r&Zconnect_default_loggerZ request_argsZ hash_checksumZ hash_algousernamepasswordrrrrrNsv      0 zSmartDL.__init__cCsdj|j|jS)NzSmartDL(r"{}", dest=r"{}"))rr.r:)rrrrrszSmartDL.__str__cCs dj|jS)Nz )rr.)rrrrrszSmartDL.__repr__cCs2dj||}tj|jd}d||jdd<dS)z Uses HTTP Basic Access authentication for the connection. :param username: Username. :type username: string :param password: Password. :type password: string z{}:{}zutf-8sBasic r AuthorizationN)rbase64standard_b64encodeencoder()rrYrZZ auth_stringZ base64stringrrrr)s z SmartDL.add_basic_authenticationcCsd|_||_||_dS)a Adds hash verification to the download. If hash is not correct, will try different mirrors. If all mirrors aren't passing hash verification, `HashFailedException` Exception will be raised. .. NOTE:: If downloaded file already exist on the destination, and hash matches, pySmartDL will not download it again. .. WARNING:: The hashing algorithm must be supported on your system, as documented at `hashlib documentation page `_. :param algorithm: Hashing algorithm. :type algorithm: string :param hash: Hash code. :type hash: string TN)rKrPrQ)r algorithmhashrrradd_hash_verificationszSmartDL.add_hash_verificationc Csdddg}tjj|j}tjj|j}|jjdx|D]}yd||f}tjj |f|j }tjj |}|j j d}|jxT|D]L} |j| jkr|jjd||jd} | j d d } |j| | d SqWWq8tjjk rw8Yq8Xq8Wd S) a4 Will attempt to fetch UNIX hash sums files (`SHA256SUMS`, `SHA1SUMS` or `MD5SUMS` files in the same url directory). Calls `self.add_hash_verification` if successful. Returns if a matching hash was found. :rtype: bool *New in 1.2.1* Z SHA256SUMSZSHA1SUMSZMD5SUMSzLooking for SUMS files...z%s/%s zFound a matching hash in %sZSUMS rN)r3r4rTr.r5r&r/r0requestRequestr(urlopenreadsplitcloselowerrstripraerror HTTPError) rZdefault_sums_filenamesZfolderZ orig_basenamer Zsums_urlZ sumsRequestobjdatalineZalgor`rrrfetch_hash_sumss(         zSmartDL.fetch_hash_sumsc $sVjdkstdjjjjd|dkr6j}n|_jr\jjdjtjn jjdjdk rt j j j rt jjj jjjkrjjdj d_dSjjd jjj tjjjfj}ytjj|jd }Wntjjtjjtjfk r}zjj|jrzjjd jt|jjd _jjd jjj |dSjj!t|jj|d_"d_WYdd}~XnXy2t#|j$d_%jjdjj%t j&j%Wn.t't(t)fk rjj!dd _%YnXt j*j%j+j,}|d d|d d d}t|dkrvjjdjt|t j&|njjdjt j&|d_xVt-|D]J\}}j.j/t0jj d|j|d |djdj1j2jd }qWt3j4t5j.fddt6t|Dj gj%fd_7dj7_8j7j t9_:|rRj;dddS)a, Starts the download task. Will raise `RuntimeError` if it's the object's already downloading. .. warning:: If you're using the non-blocking mode, Exceptions won't be raised. In that case, call `isSuccessful()` after the task is finished, to make sure the download succeeded. Call `get_errors()` to get the the exceptions. :param blocking: If true, calling this function will block the thread until the download finished. Default is *True*. :type blocking: bool r#z#cannot start (current status is {})z!Starting a new SmartDL operation.Nz"One URL and {} mirrors are loaded.zOne URL is loaded.zKDestination '%s' already exists, and the hash matches. No need to download.finishedzDownloading '{}' to '{}'...)r$z{} Trying next mirror...rzUsing url "{}"TzContent-LengthzContent-Length is {} ({}).z8Server did not send Content-Length. Filesize is unknown.rz+Launching {} threads (downloads {}/thread).z"Launching 1 thread (downloads {}). downloadingz.%.3dFcsg|]}jd|qS)z.%.3d)r:)ri)rrrr 8sz!SmartDL.start..)targetargs)raise_exceptions)||j|jd<nd|jkrP|jd=dS)z Limits the download transfer speed. :param speed: Speed in bytes per download per second. Negative values will not limit the speed. Default is `-1`. :type speed: int rsrlimitN)rJrrrArI)rZspeedrrr limit_speeds   zSmartDL.limit_speedcCs|jS)z Get the destination path of the downloaded file. Needed when no destination is provided to the class, and exists on a temp folder. :rtype: string )r:)rrrrget_destszSmartDL.get_destcCs(|js dS|rtj|jjS|jjS)a* Returns how much time did the download take, in seconds. Returns `-1` if the download task is not finished yet. :param human: If true, returns a human-readable formatted string. Else, returns an int type number :type human: bool :rtype: int/string r)rSrr get_dl_time)rrrrrrs zSmartDL.get_dl_timecCs(|js dS|rtj|jjS|jjS)z Get downloaded bytes counter in bytes. :param human: If true, returns a human-readable formatted string. Else, returns an int type number :type human: bool :rtype: int/string r)rSrrr)rrrrrr+s zSmartDL.get_dl_sizecCs(|js dS|rtj|jjS|jjS)z Get total download size in bytes. :param human: If true, returns a human-readable formatted string. Else, returns an int type number :type human: bool :rtype: int/string r)rSrrget_final_filesize)rrrrrr9s zSmartDL.get_final_filesizerc Cs\|jdkrtd|j|r dnd}t|j| }|dkrF|j|n|j}WdQRX|S)a Returns the downloaded data. Will raise `RuntimeError` if it's called when the download task is not finished yet. :param binary: If true, will read the data as binary. Else, will read it as text. :type binary: bool :param bytes: Number of bytes to read. Negative values will read until EOF. Default is `-1`. :type bytes: int :rtype: string rrzTThe download task must be finished in order to read the data. (current status is %s)rbrrN)rJrxopenrrg)rbinarybytesflagsfrorrrget_dataHs  $zSmartDL.get_datacCstj||jddjS)a Returns the downloaded data's hash. Will raise `RuntimeError` if it's called when the download task is not finished yet. :param algorithm: Hashing algorithm. :type algorithm: bool :rtype: string .. WARNING:: The hashing algorithm must be supported on your system, as documented at `hashlib documentation page `_. T)r)hashlibnewr hexdigest)rr_rrr get_data_hash[s zSmartDL.get_data_hashcCs|j}tj|S)a Returns the JSON in the downloaded data. Will raise `RuntimeError` if it's called when the download task is not finished yet. Will raise `json.decoder.JSONDecodeError` if the downloaded data is not valid JSON. :rtype: dict )rjsonloads)rrorrrget_jsoniszSmartDL.get_json) NTTrrNFNNNNN)N)r)N)F)Frr)rr)F)F)F)Fr%)Fr%)#rrrrrrrr)rarqr~rrrrrrrrrrrrrrrrrrrrrrrrrrrrr-s@ F # d          c CsZx|jstjdqW|jr"dS|jrZx|jD]}|jj|q4W|jt |j|j rp|jj ddS|rt |d}t dd|dD}tj||}|d|krd j||||}|jj ||j|dSd |_|jjd tj||jrV|d} tj|j| } | |jkr.|jjd n(|jj d|jttjj| t|jdS)z=Run function after thread pool is done. Run this in a thread.g?NzTask had errors. Exiting...rcSsg|]}tjj|qSr)r3r4getsize)rrrrrr sz+post_threadpool_actions..r"iznDiff between downloaded files and expected filesizes is {}B (filesize: {}, expected_filesize: {}, {} threads). combiningzCombining downloaded file.rzHash verification succeeded.zHash verification failed.ir%)donerrrLZ get_exceptionZget_exceptionsr&rrr+rMrVrysummathfabsrrJr/rZ combine_filesrKrzrPrQrr r3r4r5r`) rWrvZexpected_filesizeZ SmartDLObjexcrXZtotal_filesizeZdiffZerrMsgZ dest_pathZhash_rrrrts<          r)$r3urllib.requestr0 urllib.error urllib.parserrrr8r\rr|iorZmultiprocessing.dummydummyrFctypesrrrrrSrr__all__Z__version_mjaor__Z__version_minor__Z__version_micro__r __version__ Exceptionr rrrrrrrs8        K