Current File : //proc/353637/cwd/lib/python3/dist-packages/s3transfer/__pycache__/upload.cpython-312.pyc
�

�<�e)w��$�ddlZddlmZddlmZmZddlmZddlm	Z	m
Z
mZmZddl
mZmZmZmZGd�d�ZGd	�d
�ZGd�d�ZGd
�de�ZGd�de�ZGd�de�ZGd�de�ZGd�de�ZGd�de�Zy)�N)�BytesIO��readable�seekable)�IN_MEMORY_UPLOAD_TAG)�CompleteMultipartUploadTask�CreateMultipartUploadTask�SubmissionTask�Task)�ChunksizeAdjuster�DeferredOpenFile�
get_callbacks�get_filtered_dictc�&�eZdZdd�Zd�Zd�Zd�Zy)�AggregatedProgressCallbackc�.�||_||_d|_y)a�Aggregates progress updates for every provided progress callback

        :type callbacks: A list of functions that accepts bytes_transferred
            as a single argument
        :param callbacks: The callbacks to invoke when threshold is reached

        :type threshold: int
        :param threshold: The progress threshold in which to take the
            aggregated progress and invoke the progress callback with that
            aggregated progress total
        rN)�
_callbacks�
_threshold�_bytes_seen)�self�	callbacks�	thresholds   �3/usr/lib/python3/dist-packages/s3transfer/upload.py�__init__z#AggregatedProgressCallback.__init__!s��$���#������c��|xj|z
c_|j|jk\r|j�yy�N)rr�_trigger_callbacks)r�bytes_transferreds  r�__call__z#AggregatedProgressCallback.__call__1s7�����-�-�����t���.��#�#�%�/rc�D�|jdkDr|j�yy)z@Flushes out any progress that has not been sent to its callbacksrN)rr�rs r�flushz AggregatedProgressCallback.flush6s!�����a���#�#�%� rc�Z�|jD]}||j���d|_y)N)rr)rr)r�callbacks  rrz-AggregatedProgressCallback._trigger_callbacks;s+�����	9�H��t�'7�'7�8�	9���rN)i)�__name__�
__module__�__qualname__rr r#r�rrrr s��� &�
&�
rrc�>�eZdZdZd�Zd
d�Zdd�Zd�Zd�Zd�Z	d	�Z
y)�InterruptReadera�Wrapper that can interrupt reading using an error

    It uses a transfer coordinator to propagate an error if it notices
    that a read is being made while the file is being read from.

    :type fileobj: file-like obj
    :param fileobj: The file-like object to read from

    :type transfer_coordinator: s3transfer.futures.TransferCoordinator
    :param transfer_coordinator: The transfer coordinator to use if the
        reader needs to be interrupted.
    c� �||_||_yr)�_fileobj�_transfer_coordinator)r�fileobj�transfer_coordinators   rrzInterruptReader.__init__Os����
�%9��"rNc��|jjr|jj�|jj|�Sr)r.�	exceptionr-�read)r�amounts  rr3zInterruptReader.readSs:���%�%�/�/��,�,�6�6�6��}�}�!�!�&�)�)rc�<�|jj||�yr)r-�seek)r�where�whences   rr6zInterruptReader.seek]s���
�
���5�&�)rc�6�|jj�Sr)r-�tellr"s rr:zInterruptReader.tell`s���}�}�!�!�#�#rc�8�|jj�yr)r-�closer"s rr<zInterruptReader.closecs���
�
���rc��|Srr)r"s r�	__enter__zInterruptReader.__enter__fs���rc�$�|j�yr)r<)r�args�kwargss   r�__exit__zInterruptReader.__exit__is���
�
�rr)r)r&r'r(�__doc__rr3r6r:r<r>rBr)rrr+r+As*���:�*�*�$���rr+c�X�eZdZdZd
d�Zed��Zd�Zd�Zd�Z	d�Z
d	�Zd
�Zd�Z
d�Zy)�UploadInputManageraJBase manager class for handling various types of files for uploads

    This class is typically used for the UploadSubmissionTask class to help
    determine the following:

        * How to determine the size of the file
        * How to determine if a multipart upload is required
        * How to retrieve the body for a PutObject
        * How to retrieve the bodies for a set of UploadParts

    The answers/implementations differ for the various types of file inputs
    that may be accepted. All implementations must subclass and override
    public methods from this class.
    Nc�.�||_||_||_yr)�_osutilr.�_bandwidth_limiter)r�osutilr0�bandwidth_limiters    rrzUploadInputManager.__init__}s�����%9��"�"3��rc��td��)aDetermines if the source for the upload is compatible with manager

        :param upload_source: The source for which the upload will pull data
            from.

        :returns: True if the manager can handle the type of source specified
            otherwise returns False.
        zmust implement _is_compatible()��NotImplementedError��cls�
upload_sources  r�
is_compatiblez UploadInputManager.is_compatible�s��"�"C�D�Drc��td��)a�Whether the body it provides are stored in-memory

        :type operation_name: str
        :param operation_name: The name of the client operation that the body
            is being used for. Valid operation_names are ``put_object`` and
            ``upload_part``.

        :rtype: boolean
        :returns: True if the body returned by the manager will be stored in
            memory. False if the manager will not directly store the body in
            memory.
        z%must implement store_body_in_memory()rL�r�operation_names  r�stores_body_in_memoryz(UploadInputManager.stores_body_in_memory�s��"�"I�J�Jrc��td��)z�Provides the transfer size of an upload

        :type transfer_future: s3transfer.futures.TransferFuture
        :param transfer_future: The future associated with upload request
        z&must implement provide_transfer_size()rL�r�transfer_futures  r�provide_transfer_sizez(UploadInputManager.provide_transfer_size�s��"�"J�K�Krc��td��)a�Determines where a multipart upload is required

        :type transfer_future: s3transfer.futures.TransferFuture
        :param transfer_future: The future associated with upload request

        :type config: s3transfer.manager.TransferConfig
        :param config: The config associated to the transfer manager

        :rtype: boolean
        :returns: True, if the upload should be multipart based on
            configuration and size. False, otherwise.
        z*must implement requires_multipart_upload()rL�rrX�configs   r�requires_multipart_uploadz,UploadInputManager.requires_multipart_upload�s��"�"N�O�Orc��td��)a�Returns the body to use for PutObject

        :type transfer_future: s3transfer.futures.TransferFuture
        :param transfer_future: The future associated with upload request

        :type config: s3transfer.manager.TransferConfig
        :param config: The config associated to the transfer manager

        :rtype: s3transfer.utils.ReadFileChunk
        :returns: A ReadFileChunk including all progress callbacks
            associated with the transfer future.
        z$must implement get_put_object_body()rLrWs  r�get_put_object_bodyz&UploadInputManager.get_put_object_body�s��"�"H�I�Irc��td��)aYields the part number and body to use for each UploadPart

        :type transfer_future: s3transfer.futures.TransferFuture
        :param transfer_future: The future associated with upload request

        :type chunksize: int
        :param chunksize: The chunksize to use for this upload.

        :rtype: int, s3transfer.utils.ReadFileChunk
        :returns: Yields the part number and the ReadFileChunk including all
            progress callbacks associated with the transfer future for that
            specific yielded part.
        z)must implement yield_upload_part_bodies()rL)rrX�	chunksizes   r�yield_upload_part_bodiesz+UploadInputManager.yield_upload_part_bodies�s��"�"M�N�Nrc��t||j�}|jr(|jj||jd��}|S)NF)�enabled)r+r.rH�get_bandwith_limited_stream)rr/s  r�
_wrap_fileobjz UploadInputManager._wrap_fileobj�sM��!�'�4�+E�+E�F���"�"��-�-�I�I���3�3�U�J��G��rc�:�t|d�}|rt|�gSgS)N�progress)rr)rrXrs   r�_get_progress_callbacksz*UploadInputManager._get_progress_callbacks�s&��!�/�:�>�	��.�y�9�:�:��	rc�@�|D�cgc]}|j��c}Scc}wr)r#)r�aggregated_progress_callbacksr%s   r�_get_close_callbacksz'UploadInputManager._get_close_callbacks�s��/L�M�8����M�M��Ms�r)r&r'r(rCr�classmethodrQrUrYr]r_rbrfrirlr)rrrErEmsP��
�4�
�	E��	E�
K�L�
P�
J�O� ��NrrEc�V�eZdZdZed��Zd�Zd�Zd�Zd�Z	d�Z
d�Zd	�Zd
�Z
d�Zy)
�UploadFilenameInputManagerzUpload utility for filenamesc�"�t|t�Sr)�
isinstance�strrNs  rrQz(UploadFilenameInputManager.is_compatible�s���-��-�-rc��y)NFr)rSs  rrUz0UploadFilenameInputManager.stores_body_in_memory�s��rc��|jj|jj|jjj
��yr)�metarYrG�
get_file_size�	call_argsr/rWs  rrYz0UploadFilenameInputManager.provide_transfer_size�s:�����2�2��L�L�&�&��';�';�'E�'E�'M�'M�N�	
rc�H�|jj|jk\Sr)ru�size�multipart_thresholdr[s   rr]z4UploadFilenameInputManager.requires_multipart_upload�s���#�#�(�(�F�,F�,F�F�Frc��|j|�\}}|j|�}|j|�}|j|�}|jj
}|jj|||||��S)N�r/�
chunk_size�full_file_sizer�close_callbacks)�&_get_put_object_fileobj_with_full_sizerfrirlruryrG�#open_file_chunk_reader_from_fileobj)rrXr/�	full_sizerrrys       rr_z.UploadFilenameInputManager.get_put_object_body�s���!�H�H��
�����$�$�W�-���0�0��A�	��3�3�I�>���#�#�(�(���|�|�?�?���$��+�@�
�	
rc#�K�|jj}|j||�}td|dz�D]�}|j	|�}|j|�}||dz
z}|j
|jjj|||��\}	}
|j|	�}	|jj|	||
||��}||f����y�w)N�)�
start_byte�	part_sizer~r|)rury�_get_num_parts�rangerirl�'_get_upload_part_fileobj_with_full_sizerwr/rfrGr�)rrXrar~�	num_parts�part_numberrrr�r/r��read_file_chunks            rrbz3UploadFilenameInputManager.yield_upload_part_bodiess�����(�-�-�2�2���'�'���C�	� ��I��M�2�	/�K��4�4�_�E�I�"�7�7�	�B�O�"�k�A�o�6�J�"&�!M�!M��$�$�.�.�6�6�%�#�-�	"N�"��G�Y��(�(��1�G�#�l�l�N�N��$�(�#� /�O��O���.�.�5	/�s�CCc�J�t|||jj��}|S)N)�
open_function)r
rG�open)rr/r�s   r�_get_deferred_open_filez2UploadFilenameInputManager._get_deferred_open_file1s$��"��Z�t�|�|�/@�/@�
���rc��|jjj}|jj}|j	|d�|fS)Nr)rurwr/ryr��rrXr/rys    rr�zAUploadFilenameInputManager._get_put_object_fileobj_with_full_size7sB��!�&�&�0�0�8�8���#�#�(�(���+�+�G�Q�7��=�=rc�>�|d}|d}|j||�|fS)Nr�r~)r�)rr/rAr�r�s     rr�zBUploadFilenameInputManager._get_upload_part_fileobj_with_full_size<s/���L�)�
��+�,�	��+�+�G�Z�@�)�K�Krc�~�ttj|jjt|�z��Sr)�int�math�ceilrury�float)rrXr�s   rr�z)UploadFilenameInputManager._get_num_partsAs,���4�9�9�_�1�1�6�6��y�9I�I�J�K�KrN)r&r'r(rCrmrQrUrYr]r_rbr�r�r�r�r)rrroro�sG��&��.��.��
�
G�
�0/�>�>�
L�
Lrroc�8�eZdZdZed��Zd�Zd�Zd�Zd�Z	y)�UploadSeekableInputManagerz&Upload utility for an open file objectc�2�t|�xrt|�SrrrNs  rrQz(UploadSeekableInputManager.is_compatibleHs���
�&�B�8�M�+B�Brc��|dk(ryy)N�
put_objectFTr)rSs  rrUz0UploadSeekableInputManager.stores_body_in_memoryLs���\�)��rc��|jjj}|j�}|j	dd�|j�}|j	|�|jj||z
�y)Nr�)rurwr/r:r6rY)rrXr/�start_position�end_positions     rrYz0UploadSeekableInputManager.provide_transfer_sizeRsg��!�&�&�0�0�8�8��!��������Q����|�|�~�����^�$����2�2��>�)�	
rc�V�|j|d�}t|�t|�fS)Nr�)r3r�len)rr/rA�datas    rr�zBUploadSeekableInputManager._get_upload_part_fileobj_with_full_size_s,���|�|�F�;�/�0���t�}�c�$�i�'�'rc��|jjj}|j�|jjz}||fSr)rurwr/r:ryr�s    rr�zAUploadSeekableInputManager._get_put_object_fileobj_with_full_sizensA��!�&�&�0�0�8�8���|�|�~�� 4� 4� 9� 9�9����}�rN)
r&r'r(rCrmrQrUrYr�r�r)rrr�r�Es-��0��C��C��
�
(�rr�c�`��eZdZdZd�fd�	Zed��Zd�Zd�Zd�Z	d�Z
d�Zdd	�Zd
�Z
�xZS)
�UploadNonSeekableInputManagerz7Upload utility for a file-like object that cannot seek.c�6��t�|�|||�d|_y)Nr)�superr�
_initial_data)rrIr0rJ�	__class__s    �rrz&UploadNonSeekableInputManager.__init__ys���
����!5�7H�I� ��rc��t|�Sr)rrNs  rrQz+UploadNonSeekableInputManager.is_compatible}s
���
�&�&rc��y)NTr)rSs  rrUz3UploadNonSeekableInputManager.stores_body_in_memory�s��rc��yrr)rWs  rrYz3UploadNonSeekableInputManager.provide_transfer_size�s��	rc�0�|jj�#|jj|jk\S|jjj}|j}|j||d�|_t|j�|kryy)NFT)ruryrzrwr/�_readr�r�)rrXr\r/rs     rr]z7UploadNonSeekableInputManager.requires_multipart_upload�s������$�$�0�"�'�'�,�,��0J�0J�J�J�"�&�&�0�0�8�8���.�.�	�!�Z�Z���E�B����t�!�!�"�Y�.��rc���|j|�}|j|�}|jjj}|j|j|j�z||�}d|_|Sr)rirlrurwr/�
_wrap_datar�r3)rrXrrr/�bodys      rr_z1UploadNonSeekableInputManager.get_put_object_body�sn���0�0��A�	��3�3�I�>��!�&�&�0�0�8�8������������/��O�
��"����rc#�K�|jjj}d}	|j|�}|j	|�}|dz
}|j||�}|sy|j
|||�}d}||f���X�w)Nrr�)rurwr/rirlr�r�)	rrXra�file_objectr�rr�part_content�part_objects	         rrbz6UploadNonSeekableInputManager.yield_upload_part_bodies�s�����%�*�*�4�4�<�<������4�4�_�E�I�"�7�7�	�B�O��1��K��:�:�k�9�=�L����/�/��i���K�
 �L��{�*�*��s�A<A>c�T�t|j�dk(r|j|�S|t|j�kr'|jd|}|r|j|d|_|S|t|j�z
}|j|j|�z}|rd|_|S)a=
        Reads a specific amount of data from a stream and returns it. If there
        is any data in initial_data, that will be popped out first.

        :type fileobj: A file-like object that implements read
        :param fileobj: The stream to read from.

        :type amount: int
        :param amount: The number of bytes to read from the stream.

        :type truncate: bool
        :param truncate: Whether or not to truncate initial_data after
            reading from it.

        :return: Generator which generates part bodies from the initial data.
        rNr)r�r�r3)rr/r4�truncater��amount_to_reads      rr�z#UploadNonSeekableInputManager._read�s���$�t�!�!�"�a�'��<�<��'�'��S��+�+�,�,��%�%�g�v�.�D��%)�%7�%7���%@��"��K�
 �#�d�&8�&8�"9�9���!�!�G�L�L��$@�@���!$�D���rc��|jt|��}|jj|t	|�t	|�||��S)a�
        Wraps data with the interrupt reader and the file chunk reader.

        :type data: bytes
        :param data: The data to wrap.

        :type callbacks: list
        :param callbacks: The callbacks associated with the transfer future.

        :type close_callbacks: list
        :param close_callbacks: The callbacks to be called when closing the
            wrapper for the data.

        :return: Fully wrapped data.
        r|)rfrrGr�r�)rr�rrr/s     rr�z(UploadNonSeekableInputManager._wrap_data�sL�� �$�$�W�T�]�3���|�|�?�?���4�y��t�9��+�@�
�	
rr)T)r&r'r(rCrrmrQrUrYr]r_rbr�r��
__classcell__)r�s@rr�r�vsC���A�!��'��'���
�"�+�((�T
rr�c�N�eZdZdZgd�Zgd�Zd�Z	dd�Zd�Zd�Z	d	�Z
d
�Zd�Zy)
�UploadSubmissionTaskz.Task for submitting tasks to execute an upload)�ChecksumAlgorithm�SSECustomerKey�SSECustomerAlgorithm�SSECustomerKeyMD5�RequestPayer�ExpectedBucketOwner)r�r�r�r�r�c���tttg}|jjj
}|D]}|j
|�s�|cStdj|t|����)aoRetrieves a class for managing input for an upload based on file type

        :type transfer_future: s3transfer.futures.TransferFuture
        :param transfer_future: The transfer future for the request

        :rtype: class of UploadInputManager
        :returns: The appropriate class to use for managing a specific type of
            input for uploads.
        z&Input {} of type: {} is not supported.)
ror�r�rurwr/rQ�RuntimeError�format�type)rrX�upload_manager_resolver_chainr/�upload_manager_clss     r�_get_upload_input_manager_clsz2UploadSubmissionTask._get_upload_input_manager_clssx��
'�&�)�)
�%�"�&�&�0�0�8�8��"?�	*��!�/�/��8�)�)�	*��4�;�;���g��
�
�	
rNc��|j|�||j|�}|jj�|j	|�|j||�s|j
||||||�y|j||||||�y)a�
        :param client: The client associated with the transfer manager

        :type config: s3transfer.manager.TransferConfig
        :param config: The transfer config associated with the transfer
            manager

        :type osutil: s3transfer.utils.OSUtil
        :param osutil: The os utility associated to the transfer manager

        :type request_executor: s3transfer.futures.BoundedExecutor
        :param request_executor: The request executor associated with the
            transfer manager

        :type transfer_future: s3transfer.futures.TransferFuture
        :param transfer_future: The transfer future associated with the
            transfer request that tasks are being submitted for
        N)r�r.ruryrYr]�_submit_upload_request�_submit_multipart_request)r�clientr\rI�request_executorrXrJ�upload_input_managers        r�_submitzUploadSubmissionTask._submit.s���6 
�t�A�A�� 
�
�$�,�,�.?� A��
���$�$�,� �6�6��G�$�=�=��V�
�
�'�'���� ��$�

�
�*�*���� ��$�

rc�"�|jj}|j|d�}|jj	|t|j||j
|�|j|j|jd�d��|��y)Nr�)r�r/�bucket�key�
extra_argsT)r0�main_kwargs�is_final��tag)
rurw�_get_upload_task_tagr.�submit�
PutObjectTaskr_r�r�r�)	rr�r\rIr�rXr�rw�put_object_tags	         rr�z+UploadSubmissionTask._submit_upload_requestgs���$�(�(�2�2�	��2�2� �,�
��
	
�"�"�)�)���%)�%?�%?�$�3�G�G�'� �(�.�.�$�=�=�"+�"6�"6���
��	*�	
rc�z�|jj}|jj|t	|j||j
|j|jd����}g}	|j|j�}
|j|d�}|jj}t�}
|
j|j|�}|j||�}|D]e\}}|	j|jj|t!|j|||j
|j||
d�d|i��|����g|j#|j�}|jj|t%|j||j
|j|d�||	d�d	�
��y)N)r�r�r�r�)r0r��upload_part)r�r/r�r�r�r��	upload_id)r0r��pending_main_kwargsr�)r��partsT)r0r�r�r�)rurwr.r�r	r�r�r��_extra_upload_part_argsr�ryr�adjust_chunksize�multipart_chunksizerb�append�UploadPartTask�_extra_complete_multipart_argsr)rr�r\rIr�rXr�rw�create_multipart_future�part_futures�extra_part_args�upload_part_tagry�adjusterra�
part_iteratorr�r/�complete_multipart_extra_argss                   rr�z.UploadSubmissionTask._submit_multipart_request�s���$�(�(�2�2�	�#'�"<�"<�"C�"C��%�%)�%?�%?�$�'�.�.�$�=�=�"+�"6�"6�	�
�#
�����6�6�y�7K�7K�L���3�3� �-�
���#�#�(�(��$�&���-�-�f�.H�.H�$�O�	�,�E�E��Y�
�
�%2�	� �K������*�*�1�1�$�"�-1�-G�-G�&,�'.�&/�&6�&6�#,�=�=�+6�*9�
%�(�)@�-�
�(�!2��
�	�,)-�(K�(K�� � �)
�%�	
�"�"�)�)��'�%)�%?�%?�$�'�.�.�$�=�=�"?�	�"9�)�%��

�	
rc�.�t||j�Sr)r�UPLOAD_PART_ARGS�rr�s  rr�z,UploadSubmissionTask._extra_upload_part_args�s��!��T�-B�-B�C�Crc�.�t||j�Sr)r�COMPLETE_MULTIPART_ARGSr�s  rr�z3UploadSubmissionTask._extra_complete_multipart_args�s�� ��T�-I�-I�J�Jrc�8�d}|j|�rt}|Sr)rUr)rr�rTr�s    rr�z)UploadSubmissionTask._get_upload_task_tag�s�����5�5�n�E�&�C��
rr)
r&r'r(rCr�r�r�r�r�r�r�r�r�r)rrr�r�sD��8�����
�B�7�r!
�FT
�lD�
K�rr�c��eZdZdZd�Zy)r�z Task to do a nonmultipart uploadc�^�|5}|jd|||d�|��ddd�y#1swYyxYw)aP
        :param client: The client to use when calling PutObject
        :param fileobj: The file to upload.
        :param bucket: The name of the bucket to upload to
        :param key: The name of the key to upload to
        :param extra_args: A dictionary of any extra arguments that may be
            used in the upload.
        )�Bucket�Key�BodyNr))r�)rr�r/r�r�r�r�s       r�_mainzPutObjectTask._main�s>���	O���F���N�V��4�N�:�N�	O�	O�	O�s�#�,N�r&r'r(rCr�r)rrr�r��s��*�
Orr�c��eZdZdZd�Zy)r�z+Task to upload a part in a multipart uploadc
���|5}|jd|||||d�|��}	ddd�	d}
|
|d�}d|vr$|dj�}d|��}
|
|	vr|	|
||
<|S#1swY�=xYw)a�
        :param client: The client to use when calling PutObject
        :param fileobj: The file to upload.
        :param bucket: The name of the bucket to upload to
        :param key: The name of the key to upload to
        :param upload_id: The id of the upload
        :param part_number: The number representing the part of the multipart
            upload
        :param extra_args: A dictionary of any extra arguments that may be
            used in the upload.

        :rtype: dict
        :returns: A dictionary representing a part::

            {'Etag': etag_value, 'PartNumber': part_number}

            This value can be appended to a list to be used to complete
            the multipart upload.
        )r�r��UploadId�
PartNumberr�N�ETag)rrr��Checksumr))r��upper)rr�r/r�r�r�r�r�r��response�etag�
part_metadata�algorithm_name�checksum_members              rr�zUploadPartTask._mains���,�	��)�v�)�)����"�&����
�H�	�����!%�[�A�
��*�,�'�(;�<�B�B�D�N� (��(8�9�O��(�*�19�/�1J�
�o�.���!	�	�s�A�A!Nr�r)rrr�r��s
��5�&rr�)r��ior�s3transfer.compatrr�s3transfer.futuresr�s3transfer.tasksrr	r
r�s3transfer.utilsrr
rrrr+rEror�r�r�r�r�r)rr�<module>rs�����0�3�������B)�)�XxN�xN�vZL�!3�ZL�z.�!;�.�bG
�$6�G
�Tl�>�l�^
O�D�
O� )�T�)r