ikan Uploader :
Directory : /opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/
Upload File : |
| Current File : //opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/storage.cpython-37.pyc |
B
g��d � @ s� d Z ddlZddlmZ ddlmZmZmZmZ ddl Z ddl m
Z
m Z m Z m
Z
mZmZ ddlmZmZ eG dd� d��Zee d �d
d �Zd d
� Zeeee f d �dd�ZdS )zF
Represents storage where ssa data is collected, stored and extracted
� N)� dataclass)�List�Iterator�Tuple�Dict)�func�cast�case�literal_column�distinct�text)�
session_scope�
RequestResultc @ s2 e Zd ZU eed< ee ed< eed< eed< dS )�
DomainData� domain_name�domain_total_reqs�is_a_wordpress_domain� urls_numberN)�__name__�
__module__� __qualname__�str�__annotations__r �int�bool� r r �D/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/storage.pyr s
r )�returnc c s� t | ���}|�tjt�dtj�t�tj�t� tj
�t� t tj
����tjt�dtj���tjt�dtj��}tj|dd� d�}xf|D ]^\}}t|�}d}dgd }x,|D ]$\} }
} } }
| |t|
�< t ||
�}q�W t||| |d�V q�W W dQ R X dS ) z7
Iterates data from database domain-by-domain.
z%Hc S s | d S )Nr r )�itemr r r �<lambda>/ � z#iter_domains_data.<locals>.<lambda>)�keyr � )r r r r N)r
�queryr �domainr �strftime�
created_at�Count�id�max� wordpress�countr �path�group_by�order_by� itertools�groupby�tupler r )�engine�dbZresults_by_hourZresults_by_hour_groupedr �groupZdomain_results_by_hourr Zrequests_number_by_hour�_�hourZ requests_numZ is_wordpressZurlsr r r �iter_domains_data s0
r7 c c s� t | ���x}|�tjt�dtj�t�ttj t
j ��� d�t�
tj�� d�t�ttjt
j ��� d���tj|k��ttj�|�jjddid�j���tjt�dtj���tjt�dtj��}d}d gd
d gd
d gd
}}}x�|D ]~\} }
} } }
|�r<|| k�r<|t||||d �fV d gd
d gd
d gd
}}}| |t|
�< | |t|
�< |
|t|
�< | }q�W | t| |||d �fV W dQ R X dS ) z6
Iterates urls data from database url-by-url.
z%H�url_throttled_reqs�url_total_reqs�
url_slow_reqs�
literal_bindsT)�compile_kwargsNr r" )r, r8 r9 r: )r
r# r r, r r% r& ZSumr �hitting_limits�
sqlalchemy�Integer�labelr' r( �is_slow_request�filterr$ r �in_�
expression�compile�stringr- r. �dictr )r2 r Z all_pathsr3 � urls_dataZ
previous_pathr8 r9 r: r, r6 Zurl_throttled_reqZ
url_total_reqZ url_slow_reqr r r �iter_urls_dataC sH "" " rI c c sv t | ��d}|�tjtj��tj|k��tj�}t� t
|�dd� �}x$|D ] \}}|dd� |D �fV qHW W dQ R X dS )zA
Get information about durations of requests url-by-url.
c S s | d S )Nr r )r r r r r � r z#get_url_durations.<locals>.<lambda>c S s g | ] \}}|�qS r r )�.0r5 �durationr r r �
<listcomp>� s z%get_url_durations.<locals>.<listcomp>N) r
r# r r, rK rB r$ r. r/ r0 �list)r2 r r3 rH Zdurations_by_pathr! r4 r r r �get_url_durations{ s
rN )�__doc__r/ � dataclassesr �typingr r r r r> r r r r
r r �ssa.dbr
r r r7 rI r r rN r r r r �<module> s '8
........