U
Tr`1 @ s d Z ddlZddlZddlZddlZddlZddlZddlZddl m
Z
ddl m Z m
Z
ddlZdad'ddZd(d
d Zd)ddZd*ddZd+ddZd,ddZdd Zdd Zd-dd Zd d Zd d Zd!d" ZG d#d$ d$e Z G d%d& d&e
j Z dS ).z?
The Utils class contains many functions for project-wide use.
N)futures)logceilF @ c C s t | dkr t| d | ndt|dT}| D ]H}t|d*}||}|rb|| ||}qHW 5 Q R X t| q.W 5 Q R X dS )z
Combines files.
:param parts: Source files.
:type parts: list of strings
:param dest: Destination file.
:type dest: string
:param chunkSize: Fetching chunk size.
:type chunkSize: int
r wbrbN)lenshutilmoveopenreadwriteosremove)partsdestZ chunkSizeoutputpartinputdata r F/mnt/cestravail/FreezeDetection/sentSatProg_v.0.1.3/pySmartDL/utils.py
combine_files s
r utf-8c C sH t j| \}}}}}t j|d}t j|d}t j|||||fS )u
Sometimes you get an URL by a user that just isn't a real
URL because it contains unsafe characters like ' ' and so on. This
function can fix some of the problems in a similar way browsers
handle data entered by the user:
>>> url_fix(u'http://de.wikipedia.org/wiki/Elf (Begriffsklärung)')
'http://de.wikipedia.org/wiki/Elf%20%28Begriffskl%C3%A4rung%29'
:param s: Url address.
:type s: string
:param charset: The target charset for the URL if the url was
given as unicode string. Default is 'utf-8'.
:type charset: string
:rtype: string
(taken from `werkzeug.utils `_)
z/%z:&=)urllibparseurlsplitquote
quote_plus
urlunsplit)scharsetschemenetlocpathqsanchorr r r url_fix) s r( -# c C sT |d8 }| dk rd} | dkr d} d|d t | | |d |t | | d S )a:
Returns a textual progress bar.
>>> progress_bar(0.6)
'[##########--------]'
:param progress: Number between 0 and 1 describes the progress.
:type progress: float
:param length: The length of the progress bar in chars. Default is 20.
:type length: int
:rtype: string
r r [])int)progress look_feellengthr r r progress_barA s
r4 c C sb | dd} t| }|s dS ddi}tjj| |d}tjj||d}t|jd }| ||kS ) a_
Checks if a server allows `Byte serving `_,
using the Range HTTP request header and the Accept-Ranges and Content-Range HTTP response headers.
:param url: Url address.
:type url: string
:param timeout: Timeout in seconds. Default is 15.
:type timeout: int
:rtype: bool
z%20FRangez bytes=0-3)headerstimeoutContent-Length) replace get_filesizer requestRequesturlopenr0 r8 close)urlr: Zfullsizer8 requrlObjfilesizer r r is_HTTPRange_supportedU s rF c C sL t jj| f|}z t jj||d}|jd dkW S tk
rF Y dS X d S )Nr9 z
accept-rangesbytesF)r r> r? r@ r8 Exception)rB r8 r: rC rD r r r auto_detect_HTTPRange_supportn s rI c C sP z"t jj| |d}t|jd }W n( tttt jj t jj
fk
rJ Y dS X |S )z
Fetches file's size of a file over HTTP.
:param url: Url address.
:type url: string
:param timeout: Timeout in seconds. Default is 15.
:type timeout: int
:returns: Size in bytes.
:rtype: int
r9 r; r ) r r> r@ r0 r8
IndexErrorKeyError TypeErrorerror HTTPErrorURLError)rB r: rD file_sizer r r r= v s r= c C s6 ddddddddd d
d d d
dddddddg} t | S )z
Returns a random popular user-agent.
Taken from `here `_, last updated on 2019/07/26.
:returns: user-agent
:rtype: string
zsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36zsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36zyMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0zwMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15zNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0zrMozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36ziMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zyMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36zLMozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0zrMozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36zRMozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0zyMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36zMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134zMMozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0zAMozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0zMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763zRMozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0zLMozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0)randomchoice)lr r r get_random_useragent s, rT c C s t tddddddgdddd d d g}| dkrzttt| d
t|d }t| d
| }|| \}}d | }|||S | dkrd S | dkrd
S dS )a2
Human-readable formatting for filesizes. Taken from `here `_.
>>> sizeof_human(175799789)
'167.7 MB'
:param num: Size in bytes.
:type num: int
:rtype: string
BZkBMBZGBZTBZPBr r r- i z {:,.%sf} {}z0 bytesz1 byteN)listzipminr0 r r floatformat)numZ unit_listexponentquotientunitZ num_decimals
format_stringr r r sizeof_human s & ra c C s t | d d }t | } | dkr8|r,|dkr8|r4dS dS ddddd d
d g}|r`d d
ddddg}ndddddddg}g }tt|d ddD ]B}| || }|dkr|||| d| f | ||| 8 } q|r|dkr|||rdndf |r
d d d |D S d d d |D S )!az
Human-readable formatting for timing. Based on code from `here `_.
>>> time_human(175799789)
'6 years, 2 weeks, 4 days, 17 hours, 16 minutes, 29 seconds'
>>> time_human(589, fmt_short=True)
'9m49s'
:param duration: Duration in seconds.
:type duration: int/float
:param fmt_short: Format as a short string (`47s` instead of `47 seconds`)
:type fmt_short: bool
:param show_ms: Specify milliseconds in the string.
:type show_ms: bool
:rtype: string
r i r Z0sz 0 seconds< i iQ i: i $ i ssmmhhddZwwyy)secondseconds)minuteminutes)hourhours)daydays)weekweeks)monthmonths)yearZyearsms milliseconds c S s g | ] }d | qS )z%s%sr .0xr r r
s z time_human..z, c S s g | ] }d | qS )z%s %sr ry r r r r| s )r0 ranger appendjoin) Zduration fmt_shortZshow_msrv Z INTERVALSZNAMESresultiar r r
time_human s4
r c C sP t | }d}t|d*}||}|r>|| ||}q$W 5 Q R X | S )ah
Calculates a file's hash.
.. WARNING::
The hashing algorithm must be supported on your system, as documented at `hashlib documentation page `_.
:param algorithm: Hashing algorithm.
:type algorithm: string
:param path: The file path
:type path: string
:rtype: string
i r )hashlibnewr r
update hexdigest) algorithmr% ZhashAlgZblock_szfr r r r
get_file_hash s
r c C s | s
dgS t | | |k r,|dkr,|d8 }q
g }d}t | | }t|D ]>}|}|| }|| d krl| d }|||f ||d 7 }qH|S )a+
Calculates the byte chunks to download.
:param filesize: filesize in bytes.
:type filesize: int
:param threads: Number of trheads
:type threads: int
:param minChunkFile: Minimum chunk size
:type minChunkFile: int
:rtype: Array of (startByte,endByte) tuples
)r r r r )r r} r~ ) rE threads minChunkFileargsposchunkr Z startByteZendByter r r calc_chunk_size s
r c C sP t d} tsL| t j t }|t j |t d | | da| S )zg
Creates a debugging logger that prints to console.
:rtype: `logging.Logger` instance
pySmartDLzA[%(levelname)s||%(thread)d@{%(pathname)s:%(lineno)d}] %(message)sT) logging getLoggerDEFAULT_LOGGER_CREATEDsetLevelDEBUG
StreamHandler setFormatter Formatter
addHandler)Zt_logconsoler r r create_debugging_logger. s
r c @ s( e Zd ZdZdd Zdd Zdd ZdS ) DummyLoggerzk
A dummy logger. You can call `debug()`, `warning()`, etc on this object, and nothing will happen.
c C s d S Nr selfr r r __init__F s zDummyLogger.__init__c O s d S r r )r r kargsr r r
dummy_funcI s zDummyLogger.dummy_funcc C s | drt|S | jS )N__)
startswithobject __getattr__r )r namer r r r L s
zDummyLogger.__getattr__N)__name__
__module__ __qualname____doc__r r r r r r r r B s r c s@ e Zd ZdZdd Z fddZdd Zdd Zd
d Z Z S ) ManagedThreadPoolExecutorzF
Managed Thread Pool Executor. A subclass of ThreadPoolExecutor.
c C s t j| | g | _d S r )r ThreadPoolExecutorr _futures)r max_workersr r r r U s z"ManagedThreadPoolExecutor.__init__c s$ t j|f||}| j| |S r )supersubmitr r~ )r fnr kwargsfuture __class__r r r Y s z ManagedThreadPoolExecutor.submitc C s t dd | jD S )Nc S s g | ] }| qS r )donery r r r r| _ s z2ManagedThreadPoolExecutor.done..)allr r r r r r ^ s z ManagedThreadPoolExecutor.donec C s* g }| j D ]}| r
|| q
|S )zY
Return all the exceptions raised.
:rtype: List of `Exception` instances)r exceptionr~ )r rS r{ r r r get_exceptionsa s
z(ManagedThreadPoolExecutor.get_exceptionsc C s$ | j D ]}| r| S qdS )z
Returns only the first exception. Returns None if no exception was raised.
:rtype: `Exception` instance
N)r r )r r{ r r r
get_exceptionl s
z'ManagedThreadPoolExecutor.get_exception)
r r r r r r r r r
__classcell__r r r r r Q s r )r )r )r) r, )r5 )Nr5 )r5 )FF) r r urllib.requestr urllib.parse urllib.errorrQ r r
concurrentr mathr r r
r r r( r4 rF rI r= rT ra r r r r r r r r r r r r s.
3