3
]1 @ s d Z ddlZddlZddlZddlZddlZddlZddlZddl m
Z
ddl m Z m
Z
ddlZdad+dd Zd,d d Zd.ddZd/ddZd0ddZd1ddZdd Zdd Zd2d d Zd d Zd!d" Zd#d$ ZG d%d& d&e Z G d'd( d(e
j Z dS )3z?
The Utils class contains many functions for project-wide use.
N)futures)logceilFi c C s t | dkr tj| d | nlt|d\}xT| D ]L}t|d.}|j|}x |rf|j| |j|}qLW W dQ R X tj| q0W W dQ R X dS )z
Combines files.
:param parts: Source files.
:type parts: list of strings
:param dest: Destination file.
:type dest: string
:param chunkSize: Fetching chunk size.
:type chunkSize: int
r wbrbN)lenshutilmoveopenreadwriteosremove)partsdestZ chunkSizeoutputpartinputdata r =C:\Users\ibrahim.fayad\Desktop\SentinelTUI\pySmartDL\utils.py
combine_files s
r utf-8c C sH t jj| \}}}}}t jj|d}t jj|d}t jj|||||fS )u
Sometimes you get an URL by a user that just isn't a real
URL because it contains unsafe characters like ' ' and so on. This
function can fix some of the problems in a similar way browsers
handle data entered by the user:
>>> url_fix(u'http://de.wikipedia.org/wiki/Elf (Begriffsklärung)')
'http://de.wikipedia.org/wiki/Elf%20%28Begriffskl%C3%A4rung%29'
:param s: Url address.
:type s: string
:param charset: The target charset for the URL if the url was
given as unicode string. Default is 'utf-8'.
:type charset: string
:rtype: string
(taken from `werkzeug.utils `_)
z/%z:&=)urllibparseurlsplitquote
quote_plus
urlunsplit)scharsetschemenetlocpathqsZanchorr r r url_fix) s r' -# c C sT |d8 }| dk rd} | dkr d} d|d t | | |d |t | | d S )a:
Returns a textual progress bar.
>>> progress_bar(0.6)
'[##########--------]'
:param progress: Number between 0 and 1 describes the progress.
:type progress: float
:param length: The length of the progress bar in chars. Default is 20.
:type length: int
:rtype: string
r r [])int)progress look_feellengthr r r progress_barA s
r2 c C sb | j dd} t| }|s dS ddi}tjj| |d}tjj||d}t|jd }|j ||kS ) a_
Checks if a server allows `Byte serving `_,
using the Range HTTP request header and the Accept-Ranges and Content-Range HTTP response headers.
:param url: Url address.
:type url: string
:param timeout: Timeout in seconds. Default is 15.
:type timeout: int
:rtype: bool
z%20FRangez bytes=0-3)headers)timeoutzContent-Length) replace get_filesizer requestRequesturlopenr. r6 close)urlr7 Zfullsizer6 requrlObjfilesizer r r is_HTTPRange_supportedU s rB c C sH t jj| f|}y t jj||d}|jd dkS tk
rB dS X d S )N)r7 z
accept-rangesbytesF)r r: r; r< r6 Exception)r> r6 r7 r? r@ r r r auto_detect_HTTPRange_supportn s rE c C sN y"t jj| |d}t|jd }W n& tttt jj t jj
fk
rH dS X |S )z
Fetches file's size of a file over HTTP.
:param url: Url address.
:type url: string
:param timeout: Timeout in seconds. Default is 15.
:type timeout: int
:returns: Size in bytes.
:rtype: int
)r7 zContent-Lengthr ) r r: r< r. r6
IndexErrorKeyError TypeErrorerror HTTPErrorURLError)r> r7 r@ file_sizer r r r9 v s r9 c C s6 ddddddddd d
d d d
dddddddg} t j| S )z
Returns a random popular user-agent.
Taken from `here `_, last updated on 2019/07/26.
:returns: user-agent
:rtype: string
zsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36zsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36zyMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0zwMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15zNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0zrMozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36ziMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zyMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36zLMozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0zrMozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36zRMozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0zyMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134zMMozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0zAMozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0zMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763zRMozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0zLMozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0)randomchoice)lr r r get_random_useragent s* rP c C s t tddddddgdddd d d g}| dkrzttt| d
t|d }t| d
| }|| \}}d | }|j||S | dkrd S | dkrd
S dS )a2
Human-readable formatting for filesizes. Taken from `here `_.
>>> sizeof_human(175799789)
'167.7 MB'
:param num: Size in bytes.
:type num: int
:rtype: string
BZkBMBZGBZTBZPBr r r+ i z {:,.%sf} {}z0 bytesz1 byteN)listzipminr. r r floatformat)numZ unit_listexponentZquotientunitZ num_decimals
format_stringr r r sizeof_human s & r\ c C s* t | d d }t | } | dkr:| s.|dkr:|r6dS dS ddddd d
d g}|rbd(d)d*d+d,d-g}nd.d/d0d1d2d3d4g}g }xZtt|d d5d6D ]B}| || }|dkr|j||| d| f | ||| 8 } qW | r|dk r|j|| rd!nd"f |rd#jd$d% |D S d&jd'd% |D S )7az
Human-readable formatting for timing. Based on code from `here `_.
>>> time_human(175799789)
'6 years, 2 weeks, 4 days, 17 hours, 16 minutes, 29 seconds'
>>> time_human(589, fmt_short=True)
'9m49s'
:param duration: Duration in seconds.
:type duration: int/float
:param fmt_short: Format as a short string (`47s` instead of `47 seconds`)
:type fmt_short: bool
:param show_ms: Specify milliseconds in the string.
:type show_ms: bool
:rtype: string
r i r Z0sz 0 seconds< i iQ i: i $ i r! r+ mhdwysecondsecondsminuteminuteshourhoursdaydaysweekweeksmonthmonthsyearyearsms milliseconds c S s g | ] }d | qS )z%s%sr ).0xr r r
s z time_human..z, c S s g | ] }d | qS )z%s %sr )rt ru r r r rv s ssmmhhddZwwyy)rc rd )re rf )rg rh )ri rj )rk rl )rm rn )ro rp r| )r. ranger appendjoin) Zduration fmt_shortZshow_msrq Z INTERVALSZNAMESresultiar r r
time_human s2 r c C sT t j| }d}t|d.}|j|}x |r@|j| |j|}q&W W dQ R X |j S )ah
Calculates a file's hash.
.. WARNING::
The hashing algorithm must be supported on your system, as documented at `hashlib documentation page `_.
:param algorithm: Hashing algorithm.
:type algorithm: string
:param path: The file path
:type path: string
:rtype: string
r i r+ r Ni i )hashlibnewr r
update hexdigest) algorithmr% ZhashAlgZblock_szfr r r r
get_file_hash s
r c C s | s
dgS x$t | | |k r.|dkr.|d8 }q W g }d}t | | }xJt|D ]>}|}|| }|| d krr| d }|j||f ||d 7 }qNW |S )a+
Calculates the byte chunks to download.
:param filesize: filesize in bytes.
:type filesize: int
:param threads: Number of trheads
:type threads: int
:param minChunkFile: Minimum chunk size
:type minChunkFile: int
:rtype: Array of (startByte,endByte) tuples
r r )r r )r r} r~ ) rA threads minChunkFileargsposchunkr Z startByteZendByter r r calc_chunk_size s r c C sP t jd} tsL| jt j t j }|jt j |jt jd | j| da| S )zg
Creates a debugging logger that prints to console.
:rtype: `logging.Logger` instance
pySmartDLzA[%(levelname)s||%(thread)d@{%(pathname)s:%(lineno)d}] %(message)sT) logging getLoggerDEFAULT_LOGGER_CREATEDsetLevelDEBUG
StreamHandler setFormatter Formatter
addHandler)Zt_logZconsoler r r create_debugging_logger. s
r c @ s( e Zd ZdZdd Zdd Zdd ZdS ) DummyLoggerzk
A dummy logger. You can call `debug()`, `warning()`, etc on this object, and nothing will happen.
c C s d S )Nr )selfr r r __init__F s zDummyLogger.__init__c O s d S )Nr )r r Zkargsr r r
dummy_funcI s zDummyLogger.dummy_funcc C s |j drtj|S | jS )N__)
startswithobject __getattr__r )r namer r r r L s
zDummyLogger.__getattr__N)__name__
__module__ __qualname____doc__r r r r r r r r B s r c s@ e Zd ZdZdd Z fddZdd Zdd Zd
d Z Z S ) ManagedThreadPoolExecutorzF
Managed Thread Pool Executor. A subclass of ThreadPoolExecutor.
c C s t jj| | g | _d S )N)r ThreadPoolExecutorr _futures)r Z max_workersr r r r U s z"ManagedThreadPoolExecutor.__init__c s$ t j|f||}| jj| |S )N)supersubmitr r~ )r fnr kwargsfuture) __class__r r r Y s z ManagedThreadPoolExecutor.submitc C s t dd | jD S )Nc S s g | ] }|j qS r )done)rt ru r r r rv _ s z2ManagedThreadPoolExecutor.done..)allr )r r r r r ^ s z ManagedThreadPoolExecutor.donec C s. g }x$| j D ]}|j r |j|j q W |S )zY
Return all the exceptions raised.
:rtype: List of `Exception` instances)r exceptionr~ )r rO ru r r r get_exceptionsa s
z(ManagedThreadPoolExecutor.get_exceptionsc C s$ x | j D ]}|j r|j S qW dS )z
Returns only the first exception. Returns None if no exception was raised.
:rtype: `Exception` instance
N)r r )r ru r r r
get_exceptionl s z'ManagedThreadPoolExecutor.get_exception)
r r r r r r r r r
__classcell__r r )r r r Q s r i @ )r )r r( r) )r r* )r3 )Nr3 )r3 )FF) r r urllib.requestr urllib.parse urllib.errorrM r r Z
concurrentr mathr r r
r r r' r2 rB rE r9 rP r\ r r r r r r r r r r r r s.
3