o 2jh@sdZddlZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZddlZddlmZd d lmZmZmZmZmZmZmZmZd d lmZmZd d lmZm Z m!Z!m"Z"m#Z#m$Z$d dl%m&Z&e$ddZ'e ddGdddZ(e)dZ*hdZ+hdZ,hdZ-e)dZ.dddddZ/iZ0e1e-D]Z2e2e0e2<e2de0e2d<e2de0e2d <qe d!e0e3e4d"Z5e ddGd#d$d$Z6e7eedBfZ8e9ee:e3fZ;d%eee:efd&e6fd'd(Zd.e3d&e3fd/d0Z?d+ed1e@d&e7eAe8e;ffd2d3ZBd4e3d5e;d&eeedBeCge@ffd6d7ZDd8eCdBd9e3d:eCd&e@fd;d<ZEd=e3d&e7ee8ge7ed>ffe@ffd?d@ZFd+edAeAe3d&e(fdBdCZGe'jHdDgdEdF  G   dtdHe e3dBejIdIdfdJe e@ejIdKdfdLe eAe3dBejIdMdNdOdfdPe e5dBejIdQdfdRe eJdBejIdSdfd&df dTdUZKe'jHgdVdF  G GdudAe eAe3ejLdWdfdHe e3dBejIdIdfdXe e@ejIdYdZd[dfd\e e@ejId]dfd&df d^d_ZMe'jHd`dagdF  G GdudHe e3dBejIdIdfdXe e@ejIdYdZd[dfd\e e@ejId]dfd&dfdbdcZNe'jHgdddFe jOddddGdGdfdeedfe dge!dHe e3dBejIdhdfdie e3dBejIdjdfdke e@ejIdldmdfdne e@ejIdodpdfdqe"d&dfdrdsZPdS)vzHContains the 'hf cache' command group with cache management subcommands.N) defaultdict)CallableMapping) dataclass)Enum) AnnotatedAny)CLIError)ANSICachedRepoInfoCachedRevisionInfo CacheNotFound HFCacheInfo _format_size parse_hf_uriscan_cache_dir)parse_duration parse_size) RepoIdArg RepoTypeOpt RevisionOptTokenOpt get_hf_api typer_factory)outzManage local cache directory.)helpT)frozenc@s>eZdZUeeed<eeeefed<e edfed<dS)_DeletionResolution revisionsselected.missingN) __name__ __module__ __qualname__ frozensetstr__annotations__dictr r tupler+r+T/var/www/html/whisper/venv/lib/python3.10/site-packages/huggingface_hub/cli/cache.pyr1s rzA^(?P[a-zA-Z_]+)\s*(?P==|!=|>=|<=|>|<|=)\s*(?P.+)$><=>!=<=>=>refssizetypeaccessedmodified>namer4r6r7z.^(?P[a-zA-Z_]+)(?::(?Pasc|desc))?$descasc)r6r7r4r8z:asc_ascz:desc_desc SortOptions)r5modulec@s*eZdZUdZeed<eed<eed<dS)CacheDeletionCountsz>Simple counters summarizing cache deletions for CLI messaging. repo_countpartial_revision_counttotal_revision_countN)r#r$r%__doc__intr(r+r+r+r,r?Ps  r?selected_by_reporeturncCsdd}d}d}|D]\}}|t|7}t|t|jkr'|d7}|t|7}q ||}t|||S)z(Summarize deletions across repositories.rr)itemslenr r?)rEr@total_revisionsrevisions_in_full_reposrepor rAr+r+r,summarize_deletions]s   rLc Cst|dddD]P}|jd|j}t||ddd}t|t|jkr2td|dq td|d|D]}d t|j pHd }td |j d |d |j q=q dS)zBPretty-print selected cache revisions during confirmation prompts.cS|j|jfSN repo_typerepo_idlowerrKr+r+r,qz0print_cache_selected_revisions..key/cS|jSrN commit_hashrevr+r+r,rTs - z (entire repo): z (detached)z z [z] N) sortedkeysrPrQrHr rtextjoinr3r[size_on_disk_str)rErKrepo_keyr revisionr3r+r+r,print_cache_selected_revisionsos"ri hf_cache_infocCsLi}i}|jD]}|j}|||<|jD] }||f||j<qq||fS)zPCreate lookup tables so CLI commands can resolve repo ids and revisions quickly.)reposcache_idrRr r[)rj repo_lookuprevision_lookuprKrgrhr+r+r,build_cache_index~s   rotargetcCsN|ds|St|}|jstd|jdus|jrtd|jd|jS)zCReturn the cache id matching a repo target passed to `hf cache rm`.zhf://z:Only repository hf:// URIs are supported by `hf cache rm`.NzBOnly repo-level hf:// URIs are supported by `hf cache rm` for now.rX) startswithris_repor rh path_in_repor5id)rpurir+r+r,_repo_cache_id_from_targets rvinclude_revisionscCsg}i}t|jddd}|D]+}tdd|jD||<|r3t|jdddD] }|||fq(q||dfq|rI|jddd||fS|jd dd||fS) z;Flatten cache metadata into rows consumed by `hf cache ls`.cSrMrNrOrSr+r+r,rTrUz'collect_cache_entries..rVcSsh|] }|jD]}|qqSr+)r3).0rhrefr+r+r, sz(collect_cache_entries..cSrYrNrZr\r+r+r,rTr^NcSs&|dj|ddur|djfSdfS)Nrr)rlr[entryr+r+r,rTs cSs |djS)Nr)rlr|r+r+r,rTs )rbrkr&r appendsort)rjrwentries repo_refs_map sorted_reposrKrhr+r+r,collect_cache_entriess"rexprrcst|}|std|d|d|d|dtvr8tdd|dttd tvrLtd d |dttd d kr[t fd dSdvrzt dt dt dBdt dtffdd }|Sdkrdkrtdddt dt dBdt dtffdd }|Sdkrtdd dt dt dBdt dtffdd }|S) zqConvert a `hf cache ls` filter expression into the yes/no test we apply to each cache entry before displaying it.zInvalid filter expression: 'z'.rWopvaluezUnsupported operator 'z ' in filter ''. Must be one of .zUnsupported filter key '' in 'r4cs"t|dur |jS|jSrN)_compare_numeric size_on_diskrKrh_)rsize_thresholdr+r,rTs z&compile_cache_filter..>r6r7rKrhNnowrFcs>dkr|jn |dur|jn|j}|durdSt||S)Nr6F) last_accessed last_modifiedr)rKrhr timestamp)rWrsecondsr+r, _time_filtersz*compile_cache_filter.._time_filterr5r.z/Only '=' is supported for 'type' filters. Got 'rcs|jkSrN)rPrRr)expectedr+r, _type_filtersz*compile_cache_filter.._type_filterz.Only '=' is supported for 'refs' filters. Got cs2|dur|jn|t}dd|DvS)NcSsg|]}|qSr+)rR)rxryr+r+r, z>compile_cache_filter.._refs_filter..)r3getr&rR)rKrhrr3)r value_rawr+r, _refs_filtersz*compile_cache_filter.._refs_filter)_FILTER_PATTERNmatchstrip ValueErrorgrouprR_ALLOWED_OPERATORSlist _FILTER_KEYSrrr r floatbool)rrrrrrr+)rrWrrrrrr,compile_cache_filters6    & "$rleftrrightcCsT|durdS||k||k||k||k||k||kd}||vr&td|||S)z)Evaluate numeric comparisons for filters.NF)r.r0r/r-r2r1z)Unsupported numeric comparison operator: )r)rrr comparisonsr+r+r,rs r sort_expr.cst|}|std|d|d|d}tvr3tdd|dttd|r7|nt}|d k}d t d t t d fffd d }||fS)zConvert a `hf cache ls` sort expression into a key function for sorting entries. Returns: A tuple of (key_function, reverse_flag) where reverse_flag indicates whether to sort in descending order (True) or ascending order (False). zInvalid sort expression: 'z5'. Expected format: 'key' or 'key:asc' or 'key:desc'.rWorderzUnsupported sort key 'rrrr9r}rF.cs|\}}dkr|j}|fSdkr$|dur|j}|fS|j}|fSdkr8|jdur3|j}|fSd}|fSdkr]|durP|jdurK|j}|fSd}|fS|jdurX|jnd}|fStd)Nr8r4r6gr7zUnsupported sort key: )rlrRrrrr)r}rKrhrrVr+r, _sort_key s. z%compile_cache_sort.._sort_key) _SORT_PATTERNrrrRrr _SORT_KEYSr_SORT_DEFAULT_ORDER CacheEntryr*r)rrexplicit_orderrreverserr+rVr,compile_cache_sort s  rtargetscCst|\}}tt}t}g}|D]Z}|}|sq|} td| rE|| } | dur3||q| \} } ||  | | | j q|t |} | durX||q| j D]} ||  | | | j q[qdd| D}tt||t|dS)z8Resolve the deletion targets into a deletion resolution.z[0-9a-fA-F]{40}NcSsi|] \}}|t|qSr+)r&)rxrKrevsr+r+r, bsz-_resolve_deletion_targets..r r!r")rorsetrrRre fullmatchrr~addr[rvr rGrr&r*)rjrrmrnr!r r" raw_targetrploweredrrKrh matched_repofrozen_selectedr+r+r,_resolve_deletion_targetsAs@       rz list | ls)z hf cache lszhf cache ls --revisionsz*hf cache ls --filter "size>1GB" --limit 20zhf cache ls --format json)examplesF cache_dirz9Cache directory to scan (defaults to Hugging Face cache).r zCInclude revisions in the output instead of aggregated repositories.filterz-fz--filterzZFilter entries (e.g. 'size>1GB', 'type=model', 'accessed>7d'). Can be used multiple times.raSort entries by key. Supported keys: 'accessed', 'modified', 'name', 'size'. Append ':asc' or ':desc' to explicitly set the order (e.g., 'modified:asc'). Defaults: 'accessed', 'modified', 'size' default to 'desc' (newest/biggest first); 'name' defaults to 'asc' (alphabetical).limitzSLimit the number of results returned. Returns only the top N entries after sorting.c s&zt|}Wnty}z td|j|d}~ww|pg}t||d\}z fdd|D} WntyG}ztt||d}~wwt | D] fdd|D}qN|rzt |j \} } |j | | dWnty}ztt||d}~ww|dur|dkrtd |d |d|}|rd d|D} t j| gd d ddidnfdd|D} t j| gddddid|rdd|D} t| }|rtdd|D}tdd|D}ntdd| D}tdd| D}t td|d|dt|ddSdS)z&List cached repositories or revisions.Cache directory not found: N)rwcsg|]}t|qSr+)r)rxrrr+r,rszls..cs$g|]}|d|dr|qS)rrr+rxr})fnrr+r,rs$)rWrrz&Limit must be a positive integer, got rc SsFg|]\}}|dur|j|j|j|jt|j|j|jt|j dqS)N)rtrQrPrh snapshot_pathr4rr3) rlrQrPr[r'rrflast_modified_strrbr3)rxrKrhr+r+r,rs  )rtrhr4rr3rhr4r)headersid_key alignmentsc sBg|]\}}|j|j|j|j|jpd|jt|tdqS)r{)rtrQrPr4rrr3) rlrQrPrflast_accessed_strrrbrr&rxrKrrr+r,rs )rtr4rrr3rtcSsh|]\}}|qSr+r+rr+r+r,rzrzls..css |] \}}|durdVqdS)Nrr+rxrr]r+r+r, szls..css"|] \}}|dur|jVqdSrNrrr+r+r,r css|]}t|jVqdSrN)rHr rxrKr+r+r,rcs|]}|jVqdSrNrrr+r+r,rz Found z repo(s) for a total of z revision(s) and z on disk.)rrr rrrtyper BadParameterr'timerrrrtablerHsumrdr boldr)rr rrrrjexcfiltersr filter_fns sort_key_fnrrG unique_reposr@revision_count total_sizer+)rrrr,lsms/      r)zhf cache rm model/gpt2z-hf cache rm hf://models/openai-community/gpt2zhf cache rm z hf cache rm model/gpt2 --dry-runzhf cache rm model/gpt2 --yesziOne or more repo IDs (e.g. model/bert-base-uncased), repo-level hf:// URIs, or revision hashes to delete.yesz-yz--yeszSkip confirmation prompt.dry_runz,Preview deletions without removing anything.c Cszt|}Wnty}z td|j|d}~wwt||}|jr7ddd|jD}td|t |j dkrIt dt j dd |jt|j }t|j} g} | jrd| | jd | jrp| | jd | s{| | jd d | } t d | d|jdt|j|rtjdd| j| j|jddStjd|d|t|j} tjd| jd| jd|jd| j| j|jddS)z(Remove cached repositories or revisions.rN cs|]}d|VqdSr_Nr+rr+r+r,r#rzrm..zCould not find in cache: rzNothing to delete.codez repo(s)z revision(s)z and About to delete z totalling rDry run: no files were deleted.T)rrkr r4zProceed with deletion?rDeleted z repo(s) and z revision(s); freed ) repos_deletedrevisions_deletedfreed)rrr rrr"rerwarningrHr rdrExitdelete_revisionsrbrLr!r@r~rArBexpected_freed_size_strriresultconfirmexecute) rrrrrjr resolutiondetailsstrategycounts summary_parts summary_textr+r+r,rmsZ&         rzhf cache prunezhf cache prune --dry-runc CsJzt|}Wnty}z td|j|d}~wwi}t}|jD]}tdd|jD}|s3q$|||<|dd|Dq$t |dkrOt ddSt t||dd } |j t| j} t|} t d | jd | jd t||rt jd d| j| jddSt jd|d| t jd| jd| jd| j| jddS)z)Remove detached revisions from the cache.rNcss"|] }t|jdkr|VqdS)rN)rHr3rxrhr+r+r,rprzprune..csrrNrZrr+r+r,rtrrz2No unreferenced revisions found. Nothing to prune.r+rrz unreferenced revision(s) (z total).rT)rr r4zProceed?rrz! unreferenced revision(s); freed r)rr)rrr rrrkr&r updaterHrrdrrrbrLrBrrirrr) rrrrjrr!r rKdetachedrrrr+r+r,prunePsV     r)zhf cache verify gpt2z)hf cache verify gpt2 --revision refs/pr/1z.hf cache verify my-dataset --repo-type datasetrQrPrhzXCache directory to use when verifying files from cache (defaults to Hugging Face cache). local_dirz?If set, verify files under this directory instead of the cache.fail_on_missing_filesz--fail-on-missing-filesz?Fail if some files exist on the remote but are missing locally.fail_on_extra_filesz--fail-on-extra-fileszLFail if some files exist locally but are not present on the remote revision.tokenc Cs|dur|durtdtjddt|d}|j|t|dr#|jnt|||||d} d} | j rGd d d | j D} t d | d } | j rm|rbd dd | j D} t d| d } n t t| j d| jr|rd dd | jD} t d| d } n t t| jd| j} | dkrtd|d|jd| d| jtj| dtjd| jd|jd|d||j| jt| ddS)aVerify checksums for a single repo revision from cache or a local directory. Examples: - Verify main revision in cache: `hf cache verify gpt2` - Verify specific revision: `hf cache verify gpt2 --revision refs/pr/1` - Verify dataset: `hf cache verify karpathy/fineweb-edu-100b-shuffle --repo-type dataset` - Verify local dir: `hf cache verify deepseek-ai/DeepSeek-OCR --local-dir /path/to/repo` NzCCannot pass both --local-dir and --cache-dir. Use one or the other.r r)r r)rQrPrhrrr rrc ss<|]}d|dd|dd|dd|dVqdS) r_pathz : expected rz ( algorithmz), got actualNr+)rxmr+r+r,rs * zverify..u<❌ Checksum verification failed for the following file(s): rcsrrr+rxpr+r+r,rru6❌ Missing files (present remotely, absent locally): zM remote file(s) are missing locally. Use --fail-on-missing-files for details.csrrr+rr+r+r,rru4❌ Extra files (present locally, absent remotely): zV local file(s) do not exist on the remote repo. Use --fail-on-extra-files for details.zVerification failed for 'z' (z) in z. Revision: z Verified z file(s) for z 'z'. All checksums match.)rQrPcheckedr )rerrorrrrverify_repo_checksumshasattrrr' mismatchesrerd missing_pathsrrH extra_paths verified_pathrhr checked_count) rQrPrhrrrrr apir exit_coderverified_locationr+r+r,verifys`0       r)NFNNN)NFF)QrCrr collectionsrcollections.abcrr dataclassesrenumrtypingrrrhuggingface_hub.errorsr utilsr r r rrrrrutils._parsingrr _cli_utilsrrrrrr_outputr cache_clircompilerrrrrr_sort_options_dictrbrWr'r#r=r?r*rr)r& RepoRefsMaprLrirorvrrrrrrrrcommandOptionrDrArgumentrrmodelrr+r+r+r,s     (            A,6,  $   R I