B
]1 @ s d Z ddlZddlZddlZddlZddlZddlZddlZddl m
Z
ddl m Z m
Z
ddlZdad'ddZd(d
d Zd)ddZd*ddZd+ddZd,ddZdd Zdd Zd-dd Zd d Zd d Zd!d" ZG d#d$ d$e Z G d%d& d&e
j Z dS ).z?
The Utils class contains many functions for project-wide use.
N)futures)logceilF @ c C s t | dkr t| d | nlt|d\}xT| D ]L}t|d.}||}x |rf|| ||}qLW W dQ R X t| q0W W dQ R X dS )z
Combines files.
:param parts: Source files.
:type parts: list of strings
:param dest: Destination file.
:type dest: string
:param chunkSize: Fetching chunk size.
:type chunkSize: int
r wbrbN)lenshutilmoveopenreadwriteosremove)partsdestZ chunkSizeoutputpartinputdata r EC:\Users\ibrahim.fayad\Desktop\sentSatProg_v.0.1.3\pySmartDL\utils.py
combine_files s
r utf-8c C sH t j| \}}}}}t j|d}t j|d}t j|||||fS )u
Sometimes you get an URL by a user that just isn't a real
URL because it contains unsafe characters like ' ' and so on. This
function can fix some of the problems in a similar way browsers
handle data entered by the user:
>>> url_fix(u'http://de.wikipedia.org/wiki/Elf (Begriffsklärung)')
'http://de.wikipedia.org/wiki/Elf%20%28Begriffskl%C3%A4rung%29'
:param s: Url address.
:type s: string
:param charset: The target charset for the URL if the url was
given as unicode string. Default is 'utf-8'.
:type charset: string
:rtype: string
(taken from `werkzeug.utils `_)
z/%z:&=)urllibparseurlsplitquote
quote_plus
urlunsplit)scharsetschemenetlocpathqsZanchorr r r url_fix) s r' -# c C sT |d8 }| dk rd} | dkr d} d|d t | | |d |t | | d S )a:
Returns a textual progress bar.
>>> progress_bar(0.6)
'[##########--------]'
:param progress: Number between 0 and 1 describes the progress.
:type progress: float
:param length: The length of the progress bar in chars. Default is 20.
:type length: int
:rtype: string
r r [])int)progress look_feellengthr r r progress_barA s
r3 c C sb | dd} t| }|s dS ddi}tjj| |d}tjj||d}t|jd }| ||kS ) a_
Checks if a server allows `Byte serving `_,
using the Range HTTP request header and the Accept-Ranges and Content-Range HTTP response headers.
:param url: Url address.
:type url: string
:param timeout: Timeout in seconds. Default is 15.
:type timeout: int
:rtype: bool
z%20FRangez bytes=0-3)headers)timeoutzContent-Length) replace get_filesizer requestRequesturlopenr/ r7 close)urlr8 Zfullsizer7 requrlObjfilesizer r r is_HTTPRange_supportedU s rC c C sH t jj| f|}y t jj||d}|jd dkS tk
rB dS X d S )N)r8 z
accept-rangesbytesF)r r; r< r= r7 Exception)r? r7 r8 r@ rA r r r auto_detect_HTTPRange_supportn s rF c C sN y"t jj| |d}t|jd }W n& tttt jj t jj
fk
rH dS X |S )z
Fetches file's size of a file over HTTP.
:param url: Url address.
:type url: string
:param timeout: Timeout in seconds. Default is 15.
:type timeout: int
:returns: Size in bytes.
:rtype: int
)r8 zContent-Lengthr ) r r; r= r/ r7
IndexErrorKeyError TypeErrorerror HTTPErrorURLError)r? r8 rA file_sizer r r r: v s r: c C s6 ddddddddd d
d d d
dddddddg} t | S )z
Returns a random popular user-agent.
Taken from `here `_, last updated on 2019/07/26.
:returns: user-agent
:rtype: string
zsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36zsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36zyMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0zwMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15zNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0zrMozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36ziMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zyMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36zLMozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0zrMozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36zRMozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0zyMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134zMMozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0zAMozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0zMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763zRMozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0zLMozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0)randomchoice)lr r r get_random_useragent s* rQ c C s t tddddddgdddd d d g}| dkrzttt| d
t|d }t| d
| }|| \}}d | }|||S | dkrd S | dkrd
S dS )a2
Human-readable formatting for filesizes. Taken from `here `_.
>>> sizeof_human(175799789)
'167.7 MB'
:param num: Size in bytes.
:type num: int
:rtype: string
BZkBMBZGBZTBZPBr r r, i z {:,.%sf} {}z0 bytesz1 byteN)listzipminr/ r r floatformat)numZ unit_listexponentZquotientunitZ num_decimals
format_stringr r r sizeof_human s & r] c C s" t | d d }t | } | dkr8|r,|dkr8|r4dS dS ddddd d
d g}|r`d d
ddddg}ndddddddg}g }xZtt|d ddD ]B}| || }|dkr|||| d| f | ||| 8 } qW |r|dkr|||rdndf |rd d d |D S d d d |D S )!az
Human-readable formatting for timing. Based on code from `here `_.
>>> time_human(175799789)
'6 years, 2 weeks, 4 days, 17 hours, 16 minutes, 29 seconds'
>>> time_human(589, fmt_short=True)
'9m49s'
:param duration: Duration in seconds.
:type duration: int/float
:param fmt_short: Format as a short string (`47s` instead of `47 seconds`)
:type fmt_short: bool
:param show_ms: Specify milliseconds in the string.
:type show_ms: bool
:rtype: string
r i r Z0sz 0 seconds< i iQ i: i $ i ssmmhhddZwwyy)secondseconds)minuteminutes)hourhours)daydays)weekweeks)monthmonths)yearZyearsms milliseconds c S s g | ] }d | qS )z%s%sr ).0xr r r
s z time_human..z, c S s g | ] }d | qS )z%s %sr )ru rv r r r rw s )r/ ranger appendjoin) Zduration fmt_shortZshow_msrr Z INTERVALSZNAMESresultiar r r
time_human s2 r c C sT t | }d}t|d.}||}x |r@|| ||}q&W W dQ R X | S )ah
Calculates a file's hash.
.. WARNING::
The hashing algorithm must be supported on your system, as documented at `hashlib documentation page `_.
:param algorithm: Hashing algorithm.
:type algorithm: string
:param path: The file path
:type path: string
:rtype: string
i r N)hashlibnewr r
update hexdigest) algorithmr% ZhashAlgZblock_szfr r r r
get_file_hash s
r c C s | s
dgS x$t | | |k r.|dkr.|d8 }q W g }d}t | | }xJt|D ]>}|}|| }|| d krr| d }|||f ||d 7 }qNW |S )a+
Calculates the byte chunks to download.
:param filesize: filesize in bytes.
:type filesize: int
:param threads: Number of trheads
:type threads: int
:param minChunkFile: Minimum chunk size
:type minChunkFile: int
:rtype: Array of (startByte,endByte) tuples
)r r r r )r rx ry ) rB threads minChunkFileargsposchunkr} Z startByteZendByter r r calc_chunk_size s r c C sP t d} tsL| t j t }|t j |t d | | da| S )zg
Creates a debugging logger that prints to console.
:rtype: `logging.Logger` instance
pySmartDLzA[%(levelname)s||%(thread)d@{%(pathname)s:%(lineno)d}] %(message)sT) logging getLoggerDEFAULT_LOGGER_CREATEDsetLevelDEBUG
StreamHandler setFormatter Formatter
addHandler)Zt_logZconsoler r r create_debugging_logger. s
r c @ s( e Zd ZdZdd Zdd Zdd ZdS ) DummyLoggerzk
A dummy logger. You can call `debug()`, `warning()`, etc on this object, and nothing will happen.
c C s d S )Nr )selfr r r __init__F s zDummyLogger.__init__c O s d S )Nr )r r Zkargsr r r
dummy_funcI s zDummyLogger.dummy_funcc C s | drt|S | jS )N__)
startswithobject __getattr__r )r namer r r r L s
zDummyLogger.__getattr__N)__name__
__module__ __qualname____doc__r r r r r r r r B s r c s@ e Zd ZdZdd Z fddZdd Zdd Zd
d Z Z S ) ManagedThreadPoolExecutorzF
Managed Thread Pool Executor. A subclass of ThreadPoolExecutor.
c C s t j| | g | _d S )N)r ThreadPoolExecutorr _futures)r Z max_workersr r r r U s z"ManagedThreadPoolExecutor.__init__c s$ t j|f||}| j| |S )N)supersubmitr ry )r fnr kwargsfuture) __class__r r r Y s z ManagedThreadPoolExecutor.submitc C s t dd | jD S )Nc S s g | ] }| qS r )done)ru rv r r r rw _ s z2ManagedThreadPoolExecutor.done..)allr )r r r r r ^ s z ManagedThreadPoolExecutor.donec C s. g }x$| j D ]}| r || q W |S )zY
Return all the exceptions raised.
:rtype: List of `Exception` instances)r exceptionry )r rP rv r r r get_exceptionsa s
z(ManagedThreadPoolExecutor.get_exceptionsc C s$ x | j D ]}| r| S qW dS )z
Returns only the first exception. Returns None if no exception was raised.
:rtype: `Exception` instance
N)r r )r rv r r r
get_exceptionl s z'ManagedThreadPoolExecutor.get_exception)
r r r r r r r r r
__classcell__r r )r r r Q s r )r )r )r( r+ )r4 )Nr4 )r4 )FF) r r urllib.requestr urllib.parse urllib.errorrN r r Z
concurrentr mathr r r
r r r' r3 rC rF r: rQ r] r r r r r r r r r r r r s.
3