o 2j!&@s,dZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z ddl m Z mZddlmZddlmZmZmZdd lmZmZdd lmZdd lmZmZmZmZmZm Z m!Z!dd l"m#Z#errdd l$m%Z%e&e'Z(dZ)dZ*e GdddZ+e GdddZ,e GdddZ-e GdddZ.e ddGdddZ/e GdddZ0e GdddZ1e Gd d!d!Z2d"e3d#efd$d%Z4d"e3d#e5fd&d'Z6e Gd(d)d)Z7e Gd*d+d+Z8Gd,d-d-Z9d.e3d#e:e;e3e3ffd/d0ZfdBfd1d2Z?d3e3d#e e;e3e=e>ffd4d5Z@d6d7d8e3d9e3d#e e;e3e=e>effd:d;ZAdd#e3fd=d>ZBdd?d"e3d@edAdBe=dCe>dDe=dEe>dFe3dGe3dHe5dIe5dJe5dKedBd#e7fdLdMZC N N N N N  dndOe3dPe3d6d7dQe5dIe5dHe5dRe5dJe5dSe9dBdTedBd#e8fdUdVZDdWe8d#dfdXdYZEdWe8dZe3d#dfd[d\ZFdZe3d#e8fd]d^ZGdodWe8d6d7d_e5dTedBd#df d`daZHdWe8d#dfdbdcZI  dpdNdNdNdNdNddddddNdNdNddddOe3dBdPe3dBd6d7dQe5dIe5dHe5dRe5dJe5dee:e3dBdfe:e3dBdge3dBdWe3dBdhe3dBdie5d_e5dje5dke5e3BdBd#e8f$dldmZJdS)qzzShared logic for bucket operations. This module contains the core buckets logic used by both the CLI and the Python API. N)Iterator) dataclassfield)datetimetimezone)Path) TYPE_CHECKINGAnyLiteral) constantslogging)BucketNotFoundError)HfUri StatusLine XetFileDatadisable_progress_barsenable_progress_barsparse_datetime parse_hf_uri)_looks_like_hf_urlHfApiz hf://buckets/c@sBeZdZUdZeed<eed<eed<eed<eed<ddZ d S) BucketInfoa Contains information about a bucket on the Hub. This object is returned by [`bucket_info`] and [`list_buckets`]. Attributes: id (`str`): ID of the bucket. private (`bool`): Is the bucket private. created_at (`datetime`): Date of creation of the bucket on the Hub. size (`int`): Size of the bucket in bytes. total_files (`int`): Total number of files in the bucket. idprivate created_atsize total_filescKsV|d|_|d|_t|d|_|d|_|d|_|jjdi|dS)Nrr createdAtr totalFiles) poprrrrrr__dict__update)selfkwargsr"r"S/var/www/html/whisper/venv/lib/python3.10/site-packages/huggingface_hub/_buckets.py__init__Vs    zBucketInfo.__init__N) __name__ __module__ __qualname____doc__str__annotations__boolrintr)r"r"r"r(r>s  rc@seZdZUeeBeBed<eed<eddZedBed<eddZ e dBed<eddZ e ed <eddZ edBed <dd d Z dS)_BucketAddFilesource destinationNdefaultxet_hashrFinitmtime content_typereturncCsxd|_t|jttfrt|jd|_|jdur"t|jd|_tt|jt s2t j |jdnt d|_dSNrr)r; isinstancer3r.r mimetypes guess_typer4r1bytesospathgetmtimetimer:r&r"r"r( __post_init__is ( z_BucketAddFile.__post_init__r<N)r*r+r,r.rrAr/rr7rr1r:r;rGr"r"r"r(r2_s r2c@sveZdZUeed<eed<eed<eed<eddZedBed<edd Zeed <edd Z edBed <dd dZ dS)_BucketCopyFiler4r7source_repo_typesource_repo_idNr5rFr8r:r;r<cCs(t|jd|_ttd|_dSr=)r?r@r4r;r1rEr:rFr"r"r(rGsz_BucketCopyFile.__post_init__rH) r*r+r,r.r/rrr1r:r;rGr"r"r"r(rIus rIc@seZdZUeed<dS)_BucketDeleteFilerCN)r*r+r,r.r/r"r"r"r(rLs rLT)frozenc@s"eZdZUdZeed<eed<dS)BucketFileMetadataa%Data structure containing information about a file in a bucket. Returned by [`get_bucket_file_metadata`]. Args: size (`int`): Size of the file in bytes. xet_file_data (`XetFileData`): Xet information for the file (hash and refresh route). r xet_file_dataN)r*r+r,r-r1r/rr"r"r"r(rNs  rNc@sfeZdZUdZeed<dZeed<eddZeed<eddZ eed<eddZ e ed <dd d Z d S) BucketUrlaDescribes a bucket URL on the Hub. `BucketUrl` is returned by [`create_bucket`]. At initialization, the URL is parsed to populate properties: - endpoint (`str`) - namespace (`str`) - bucket_id (`str`) - url (`str`) - uri (`HfUri`) Args: url (`str`): String value of the bucket url. endpoint (`str`, *optional*): Endpoint of the Hub. Defaults to . urlendpointFr8 namespace bucket_idurir<NcCs|jptj|_|j|jdd}|dr|tdd}t|}|j r.t d|j|j dd|_ |j |_||_dS)NrR/zbuckets/zUnable to parse bucket URL: r)rSr ENDPOINTrQreplacestrip startswithlen_parse_bucket_uri path_in_repo ValueErrorrsplitrTrUrV)r&url_pathparsedr"r"r(rGs  zBucketUrl.__post_init__rH) r*r+r,r-r.r/rSrrTrUrVrrGr"r"r"r(rPs  rPc@sVeZdZUdZeded<eed<eed<eed<edBed<edBed <d d Z dS) BucketFilez Contains information about a file in a bucket on the Hub. This object is returned by [`list_bucket_tree`]. Similar to [`RepoFile`] but for files in buckets. filetyperCrr7Nr: uploaded_atcKsv|d|_|d|_|d|_|d|_|dd}|r$t|nd|_|dd}|r6t||_dSd|_dS)NrerCrxetHashr: uploadedAt)r#rerCrr7rr:rf)r&r'r:rfr"r"r(r)s      zBucketFile.__init__) r*r+r,r-r r/r.r1rr)r"r"r"r(rcs     rcc@s:eZdZUdZeded<eed<edBed<ddZdS) BucketFolderz Contains information about a directory in a bucket on the Hub. This object is returned by [`list_bucket_tree`]. Similar to [`RepoFolder`] but for directories in buckets. directoryrerCNrfcKs`|d|_|d|_|ddp|dd}|r+t|tr$||_dSt||_dSd|_dS)NrerCrhrf)r#rerCr>rrrf)r&r'rfr"r"r(r)s  zBucketFolder.__init__) r*r+r,r-r r/r.rr)r"r"r"r(ris    rirCr<cCs|tjs t|rt|}|jstd|d|S|dd}t|dks/|dr/|ds7td|d|dd|d}d |dd }t d ||d S) aParse a bucket path into a HfUri. Accepts: - `hf://buckets/namespace/name(/path/in/repo)` URIs, - Hugging Face web URLs such as `https://huggingface.co/buckets/namespace/name(/tree/path)`, - plain `namespace/name(/path/in/repo)` paths. zInvalid bucket path: z*. Must be a bucket URI (hf://buckets/...).rWrr zInvalid bucket path: 'z)'. Expected format: namespace/bucket_nameNbucket)rerr^) r[r HF_PROTOCOLrr is_bucketr_r`r\joinr)rCrbpartsrUprefixr"r"r(r]s r]cCs|tjsdSt|jS)zCheck if a path is a bucket path. Do not raise if the path is not a hf:// URI. Raise if the path is a hf:// URI but with an incorrect format. F)r[r rmrrn)rCr"r"r(_is_bucket_paths  rrc@sreZdZUdZeded<eed<dZedBed<dZ eed<dZ edBed <dZ edBed <dZ e dBed <dS) SyncOperationz,Represents a sync operation to be performed.)uploaddownloaddeleteskipactionrCNrrRreason local_mtime remote_mtime bucket_file)r*r+r,r-r r/r.rr1ryrzr{r|rcr"r"r"r(rss   rsc@sZeZdZUdZeed<eed<eed<eedZee ed<de ee eBffdd Z d S) SyncPlanz Represents a complete sync plan.r3dest timestamp)default_factory operationsr<cCsttdd|jD}tdd|jD}tdd|jD}tdd|jD}tdd|jD}|||||dS)Ncs|] }|jdkrdVqdS)rtr Nrx.0opr"r"r( 5z#SyncPlan.summary..csr)rur Nrrr"r"r(r6rcsr)rvr Nrrr"r"r(r7rcsr)rwr Nrrr"r"r(r8rcss$|] }|jdvr|jp dVqdS)rtrurN)rxrrr"r"r(r9s")uploads downloadsdeletesskips total_size)sumr)r&rrrrrr"r"r(summary4szSyncPlan.summaryN) r*r+r,r-r.r/rlistrrsdictr1rr"r"r"r(r}+s r}c @s`eZdZdZ   d deedBdeedBdeeeefdBfddZded efd d Z dS) FilterMatcherz4Matches file paths against include/exclude patterns.Ninclude_patternsexclude_patterns filter_rulescCs"|pg|_|pg|_|p g|_dS)aInitialize the filter matcher. Args: include_patterns: Patterns to include (from --include) exclude_patterns: Patterns to exclude (from --exclude) filter_rules: Rules from filter file as list of ("+"/"-", pattern) tuples Nrrr)r&rrrr"r"r(r)Ks zFilterMatcher.__init__rCr<cCsr|jD]\}}t||r|dkSq|jD] }t||r"dSq|jD] }t||r1dSq&|jr7dSdS)zCheck if a path should be included based on the filter rules. Filtering rules: - Filters are evaluated in order, first matching rule decides - If no rules match, include by default (unless include patterns are specified) +FT)rfnmatchrr)r&rCsignpatternr"r"r(matches\s      zFilterMatcher.matches)NNN) r*r+r,r-rr.tupler)r0rr"r"r"r(rHs   r filter_filecCsg}t|G}|D];}|}|r|drq |dr*|d|ddfq |dr=|d|ddfq |d|fq Wd|S1sPwY|S)zParse a filter file and return a list of (sign, pattern) tuples. Filter file format: - Lines starting with "+" are include patterns - Lines starting with "-" are exclude patterns - Empty lines and lines starting with "#" are ignored #rr N-)openrZr[append)rrulesfliner"r"r(_parse_filter_fileys       rcCsDzt|}Wn tyYdSwt|jrdS|j|jdfS)zStat a local file and return (size, mtime_ms). Returns None if the path is missing or is a directory. Uses a single ``os.stat`` call so callers don't pay for multiple syscalls per file. Nr)rBstatOSErrorS_ISDIRst_modest_sizest_mtime)rCstr"r"r( _stat_locals  r local_pathccstj|}tj|std|t|D]2\}}}|D]*}tj||}t|}|dur2q tj||}| tj d}||d|dfVq qdS)zsList all files in a local directory. Yields: tuple: (relative_path, size, mtime_ms) for each file z Local path must be a directory: NrWrr ) rBrCabspathisdirr_walkrorrelpathrYsep)rroot_filesfilename full_path stat_inforel_pathr"r"r(_list_local_filess  rapirrUrqccs|j||pdddD]I}t|trq |j}|r>||dr*|t|dd}n||kr=d|vr:|dddn|}nq |}|jrJ|jdnd}||j ||fVq dS) zList all files in a bucket with a given prefix. Yields: tuple: (relative_path, size, mtime_ms, bucket_file) for each file. bucket_file is the BucketFile object from list_bucket_tree. NT)rq recursiverWr rr) list_bucket_treer>rirCr[r\rsplitr:rr)rrUrqitemrCrmtime_msr"r"r(_list_remote_filess rrcCstj|dtjdS)z3Convert mtime in milliseconds to ISO format string.r)tz)r fromtimestamprutc isoformat)rr"r"r( _mtime_to_isosr)r|rxr source_size source_mtime dest_size dest_mtimesource_newer_labeldest_newer_label ignore_sizes ignore_timesignore_existingr|c Cst|dkr|n|} t|dkr|n|} ||| | d}| r'td ddd|S||k}||tk}|rV|r@td ||| d|S||tk}|rJ|nd}td d|d|S| ro|retd |d| d|Std dd d|S|ss|r|rwdn|}td ||| d|Std dd d|S) aCompare source and dest files and return the appropriate sync operation. This is a unified helper for both upload and download directions. Args: path: Relative file path action: "upload" or "download" source_size: Size of the source file (bytes) source_mtime: Mtime of the source file (milliseconds) dest_size: Size of the destination file (bytes) dest_mtime: Mtime of the destination file (milliseconds) source_newer_label: Label when source is newer (e.g., "local newer" or "remote newer") dest_newer_label: Label when dest is newer (e.g., "remote newer" or "local newer") ignore_sizes: Only compare mtime ignore_times: Only compare size ignore_existing: Skip files that exist on receiver bucket_file: BucketFile object (for downloads only) Returns: SyncOperation describing the action to take rt)rCrrzr{rwz&exists on receiver (--ignore-existing))rxry)rxryr|z same mtimez size differsz same size identicalNr")rrs_SYNC_TIME_WINDOW_MS)rCrxrrrrrrrrrr|local_mtime_isoremote_mtime_iso base_kwargs size_differs source_newer dest_newer skip_reasonryr"r"r(_compare_files_for_syncs2$    rFr3r~rvexistingfilter_matcherstatusc %Cs|pt}t| o t|} t|ot| } | s| stdt||ttjd} d} | rt j |}t |}|j |j}}t j |sOtd|i}t|D]\}}}||re||f||<| rr| dt|dqU| r| dt|di}| rz||j} Wn tyYnwz4t|||D]+\}}}}||r||f||<| r| durd| nd}| d t||dqWntytd |d Ynw| r| d t|dt|t|B}| r| d t|d t|D]}||}||}|rG|sG|r2| j !t"d||ddt#|ddq| j !t"d||ddt#|ddq|rj|rj|\}}|\}} | j !t$|d|||| dd|||d q|s|r|r| j !t"d||ddt#|ddq| St |}|j |j}}t j |}i}i}!| rz||j} Wn tyYnwt|||D]3\}}}}"||r||f||<|"|!|<| r| durd| nd}| d t||dq| r| d t|di}t j |rY|r/t|D] \}}}||r||f||<| r,| dt|dq n*|D]'}t j %||}#t&|#}$|$durEq1|$||<| rW| dt|dq1| rg| dt|dt|t|B}| r| d t|d t|D]}||}||}|r|s|r| j !t"d||ddt#|ddq| j !t"d||ddt#|d|!|dq|r|r|\}} |\}}| j !t$|d|| ||dd||||!|d q|s |r |r | j !t"d||ddt#|ddq| S)zCompute the sync plan by comparing source and destination. Returns: SyncPlan with all operations to be performed z[One of source or dest must be a bucket path (hf://buckets/...) and the other must be local.r3r~rNzSource must be a directory: zScanning local directory (z files)rWrRzScanning remote bucket (zBucket 'z' not found, treating as empty.zComparing files (z paths)rwrznew file (--existing)r )rxrCrryrzrtznew filez local newerz remote newer) rCrxrrrrrrrrrrvznot in source (--delete))rxrCrryr{ru)rxrCrryr{r|) rCrxrrrrrrrrrr|)'rrrr_r}rnowrrrrBrCrr]rr^rrrr%r\done bucket_infor Exceptionrrloggerdebugsetkeyssortedgetrrrsrrror)%r3r~rrvrrrrrr is_upload is_downloadplan remote_totalrrbrUrq local_filesrrr remote_filesr total_str all_pathsrC local_info remote_info local_sizerz remote_sizer{bucket_file_mapr| local_filerr"r"r(_compute_sync_plan*s                  {                 rrcCsd|j|j|j|d}|t|d|jD]4}d|j|j |j d}|j dur0|j |d<|j dur:|j |d<|j durD|j |d <|t|dqdS) z1Write a sync plan as JSONL to a file-like object.header)rer3r~rr operation)rerxrCryNrrzr{)r3r~rrwritejsondumpsrrxrCryrrzr{)rrrrop_dictr"r"r( _write_plan&s*       r plan_filecCs:t|d}t||WddS1swYdS)z!Save a sync plan to a JSONL file.wN)rr)rrrr"r"r( _save_planCs  "rc Cst| }|}Wdn1swY|s!td|t|d}|ddkr3tdt|d|d|d d }|d dD].}t|}|dd krUqF|jt |d |d|d|dd|d|ddqF|S)z#Load a sync plan from a JSONL file.NzEmpty plan file: rrerz0Invalid plan file: expected header as first liner3r~rrr rrxrCrryrRrzr{)rxrCrryrzr{) r readlinesr_rloadsrr}rrrs)rrlinesrrrrr"r"r( _load_planIs8     rverbosec Cst|j o t|j}t|jot|j }|rtj|j}t|j}|j|j}} g} g} |j D]o} | j dkrgtj || j} | rM| d| jn| j}|r_t d| jd| j d| | |fq2dkr| rv| d| jn| j}|rt d| jd| j d| |q2dkr|rt d | jd| j dq2q2| s| r|rg}| r|d t| d | r|d t| d |d ||j|| pd| pdddSdS|rt|j}|j|j}} tj|j}tj|ddg}g}|j D]} | j dkrTtj || j} tjtj| dd|r-t d| jd| j d| jdur=|| j| fq| rH| d| jn| j}||| fq| j dkrytj || j} |rrt d| jd| j d|| q| j dkr|rt d | jd| j dqt|dkr|r|dt|d ||||r|r|dt|d|D]9}tj|rt|tj|}||krz t|tj|}Wn tyYnw||ksؐqdSdS)zExecute a sync plan.rtrWz Uploading: z ()rvz Deleting: rwz Skipping: z uploading z filesz deleting z, N)addrvT)exist_okruz Downloading: rz Downloading z Deleting z local files)rrr3r~rBrCrr]rr^rrxroprintryrr\r capitalizebatch_bucket_filesmakedirsdirnamer|download_bucket_filesexistsremovermdirr)rrr rrrrrbrUrq add_files delete_pathsrr remote_pathrpdownload_files delete_files file_pathparentr"r"r( _execute_plants                   *rcCsl|}td|jd|jtd|dtd|dtd|dtd |d d S) z!Print a summary of the sync plan.z Sync plan: z -> z Uploads: rz Downloads: rz Deletes: rz Skips: rN)rr r3r~)rrr"r"r(_print_plan_summarys r)rvrrrrincludeexclude filter_fromrapplydry_runr quiettokenrr r!r"r#r$r%c Cs|durddlm}||d}| r|s|rtd| dur!td|r'td|r-td|r3td |r9td | r?td | rEtd |rKtd |rQtd| rWtdt| }t| d}|skt|td|rptzt||||dW|rt n|rt ww|std|S|r|stdt |}t |}|r|rtd|s|std|r|rtd|r|rtd| r| rtd|rt j |rt j |std|n t j |std|d}| rt| }t|| |d}t| o| d}t||||||||||d }| rt|tj|S| r3t|| |s1t|td| |S|s:t||}|dd kr\|d!d kr\|d"d kr\|sZtd#|S|sctd$|ritzt||||dW|ryt n|rt ww|std|S)%aSync files between a local directory and a bucket. This is equivalent to the ``hf buckets sync`` CLI command. One of ``source`` or ``dest`` must be a bucket path (``hf://buckets/...``) and the other must be a local directory path. Args: source (`str`, *optional*): Source path: local directory or ``hf://buckets/namespace/bucket_name(/prefix)``. Required unless using ``apply``. dest (`str`, *optional*): Destination path: local directory or ``hf://buckets/namespace/bucket_name(/prefix)``. Required unless using ``apply``. api ([`HfApi`]): The HfApi instance to use for API calls. delete (`bool`, *optional*, defaults to `False`): Delete destination files not present in source. ignore_times (`bool`, *optional*, defaults to `False`): Skip files only based on size, ignoring modification times. ignore_sizes (`bool`, *optional*, defaults to `False`): Skip files only based on modification times, ignoring sizes. existing (`bool`, *optional*, defaults to `False`): Skip creating new files on receiver (only update existing files). ignore_existing (`bool`, *optional*, defaults to `False`): Skip updating files that exist on receiver (only create new files). include (`list[str]`, *optional*): Include files matching patterns (fnmatch-style). exclude (`list[str]`, *optional*): Exclude files matching patterns (fnmatch-style). filter_from (`str`, *optional*): Path to a filter file with include/exclude rules. plan (`str`, *optional*): Save sync plan to this JSONL file instead of executing. apply (`str`, *optional*): Apply a previously saved plan file. When set, ``source`` and ``dest`` are not needed. dry_run (`bool`, *optional*, defaults to `False`): Print sync plan to stdout as JSONL without executing. verbose (`bool`, *optional*, defaults to `False`): Show detailed per-file operations. quiet (`bool`, *optional*, defaults to `False`): Suppress all output and progress bars. token (Union[bool, str, None], optional): A valid user access token. If not provided, the locally saved token will be used. Returns: [`SyncPlan`]: The computed (or loaded) sync plan. Raises: `ValueError`: If arguments are invalid (e.g., both paths are remote, conflicting options). Example: ```python >>> from huggingface_hub import HfApi >>> api = HfApi() # Upload local directory to bucket >>> api.sync_bucket("./data", "hf://buckets/username/my-bucket") # Download bucket to local directory >>> api.sync_bucket("hf://buckets/username/my-bucket", "./data") # Sync with delete and filtering >>> api.sync_bucket( ... "./data", ... "hf://buckets/username/my-bucket", ... delete=True, ... include=["*.safetensors"], ... ) # Dry run: preview what would be synced >>> plan = api.sync_bucket("./data", "hf://buckets/username/my-bucket", dry_run=True) >>> plan.summary() {'uploads': 3, 'downloads': 0, 'deletes': 0, 'skips': 1, 'total_size': 4096} # Save plan for review, then apply >>> api.sync_bucket("./data", "hf://buckets/username/my-bucket", plan="sync-plan.jsonl") >>> api.sync_bucket(apply="sync-plan.jsonl") ``` Nr r)r%z,Cannot specify source/dest when using apply.z#Cannot specify both plan and apply.z'Cannot specify delete when using apply.z-Cannot specify ignore_times when using apply.z-Cannot specify ignore_sizes when using apply.z(Cannot specify include when using apply.z(Cannot specify exclude when using apply.z,Cannot specify filter_from when using apply.z)Cannot specify existing when using apply.z0Cannot specify ignore_existing when using apply.z(Cannot specify dry_run when using apply.)enabledzExecuting plan...)r rzSync completed.z7Both source and dest are required (unless using apply).z?Remote to remote sync is not supported. One path must be local.z?One of source or dest must be a bucket path (hf://buckets/...).z2Cannot specify both ignore_times and ignore_sizes.z1Cannot specify both existing and ignore_existing.z%Cannot specify both dry_run and plan.z!Destination must be a directory: z&Source must be an existing directory: r) r3r~rrvrrrrrrzPlan saved to: rrrrzNothing to sync.z Syncing...)hf_apirr_rrrr rrrrrrBrCrrrrrrsysstdoutrr)r3r~rrvrrrrrr r!rr"r#r r$r%r sync_planrsource_is_bucketdest_is_bucketrrrr"r"r(sync_bucket_internalsc      *r-)FFFFFNN)FN)NN)Kr-rrr?rBrr(rEcollections.abcr dataclassesrrrrpathlibrtypingrr r rRr r errorsrutilsrrrrrrrutils._hf_urisrr'r get_loggerr*r BUCKET_PREFIXrrr2rIrLrNrPrcrir.r]r0rrrsr}rrrrr1floatrrrrrrrrrrrr-r"r"r"r(sP    $    ' 1 *"      L   }$+d