Fedora 43 native package

python3-tokenizers

Implementation of today's most used tokenizers

Packages / Fedora 43 / Unspecified / python3-tokenizers

[Source: python-tokenizers]

Package: python3-tokenizers (0.22.2-2.fc43)

[Project overview: python-tokenizers]

External Resources:

Homepage: [github.com]

Implementation of today's most used tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Bindings over the rust-tokenizers implementation.

Other Packages Related to python3-tokenizers:

dep: (python3.14dist(huggingface-hub) < 2~~ with python3.14dist(huggingface-hub) >= 0.16.4)
Package not available
dep: ld-linux-aarch64.so.1()(64bit)
Package not available
dep: ld-linux-aarch64.so.1(GLIBC_2.17)(64bit)
Package not available
dep: libgcc_s.so.1()(64bit)
Package not available
dep: libgcc_s.so.1(GCC_3.0)(64bit)
Package not available
dep: libgcc_s.so.1(GCC_3.3)(64bit)
Package not available
dep: libgcc_s.so.1(GCC_4.2.0)(64bit)
Package not available
dep: libm.so.6()(64bit)
Package not available
dep: libm.so.6(GLIBC_2.29)(64bit)
Package not available
dep: libonig.so.5()(64bit)
Package not available
dep: libstdc++.so.6()(64bit)
Package not available
dep: libstdc++.so.6(CXXABI_1.3)(64bit)
Package not available
dep: libstdc++.so.6(GLIBCXX_3.4)(64bit)
Package not available
dep: libstdc++.so.6(GLIBCXX_3.4.30)(64bit)
Package not available
dep: python(abi) (= 3.14)
Package not available
dep: rtld(GNU_HASH)
Package not available
dep: libc.so.6(GLIBC_2.34)(64bit)
Package not available

Download python3-tokenizers

Architecture	Package Size	Installed Size	Files
aarch64	1.8 MiB	5.7 MiB	[list of files]
x86_64	2.0 MiB	6.1 MiB	[list of files]

Chemins de fichiers du paquet (94)

Paths come from the repository package-file index for the observed builds. They describe archive/package associations, not every file that will exist on a running system after maintainer scripts, alternatives, generated state, diversions, or installation choices.

/usr/lib64/python3.14/site-packages/tokenizers
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/INSTALLER
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/licenses
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/licenses/LICENSE
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/licenses/LICENSE.dependencies
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/METADATA
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/WHEEL
/usr/lib64/python3.14/site-packages/tokenizers/decoders
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations
/usr/lib64/python3.14/site-packages/tokenizers/implementations/base_tokenizer.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/bert_wordpiece.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/byte_level_bpe.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/char_level_bpe.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/sentencepiece_bpe.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/sentencepiece_unigram.py
/usr/lib64/python3.14/site-packages/tokenizers/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/models
/usr/lib64/python3.14/site-packages/tokenizers/models/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/models/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/models/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/models/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/models/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/normalizers
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/processors
/usr/lib64/python3.14/site-packages/tokenizers/processors/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/processors/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/processors/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/processors/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/processors/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tokenizers.abi3.so
/usr/lib64/python3.14/site-packages/tokenizers/tokenizers.pyi
/usr/lib64/python3.14/site-packages/tokenizers/tools
/usr/lib64/python3.14/site-packages/tokenizers/tools/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tools/visualizer.py
/usr/lib64/python3.14/site-packages/tokenizers/tools/visualizer-styles.css
/usr/lib64/python3.14/site-packages/tokenizers/trainers
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-314.pyc
/usr/lib/.build-id
/usr/lib/.build-id/58
/usr/lib/.build-id/58/4620241b3f6da5bcfcd524f636dc3ea691109d
/usr/lib/.build-id/d3
/usr/lib/.build-id/d3/65c1799eed55b4c4162de821f8faae2bf64cac
/usr/share/doc/python3-tokenizers
/usr/share/doc/python3-tokenizers/CHANGELOG.md
/usr/share/doc/python3-tokenizers/README.md
/usr/share/licenses/python3-tokenizers
/usr/share/licenses/python3-tokenizers/LICENSE
/usr/share/licenses/python3-tokenizers/LICENSE.dependencies

Field source: Fedora 43 updates aarch64 revision 43-updates-aarch64:ac6921c749da1e864e0971122151b63bb17e427965b3d9f9312a07b49ff10cac

Utiliser ce paquet

OpenFactory peut démarrer ce système d'exploitation dans une machine virtuelle du navigateur, ou lancer une construction qui inclut le nom natif du paquet de cet enregistrement.

Essayer Fedora 43 dans le navigateur Construire une image qui inclut python3-tokenizers

Versions, suites et dépôts

Chaque ligne est une métadonnée d'index pour une version, une architecture, une suite et un dépôt. Noms, URL et tailles viennent de la source ; un lien est un emplacement de récupération mutable, pas une redistribution OpenFactory.

Version	Release	Architecture	Repository	Package size	Installed size	Publisher repository artifact
0.22.2-2.fc43	43 / updates	aarch64	Fedora 43 · updates · aarch64	1.8 MiB	5.7 MiB	Packages/p/python3-tokenizers-0.22.2-2.fc43.aarch64.rpm
0.22.2-2.fc43	43 / updates	x86_64	Fedora 43 · updates · x86_64	2.0 MiB	6.1 MiB	Packages/p/python3-tokenizers-0.22.2-2.fc43.x86_64.rpm

Field source: Fedora 43 updates aarch64 revision 43-updates-aarch64:ac6921c749da1e864e0971122151b63bb17e427965b3d9f9312a07b49ff10cac

Empreintes et dates d'observation

For an APT source, signature verification authenticates the repository metadata chain and the Packages index containing this source-reported artifact digest. It does not certify package safety.

0.22.2-2.fc43 / aarch64Observed Sep 1, 2026 to Sep 1, 2026

Verification status: Metadata observed; artifact bytes were not independently fetched or hashed by this catalog import. The digest below is source-reported.

Source-reported sha256: 3c3f392886ecad521467d5a01785d256853d5aaaf08dd3ef3fa887a364e36d32

After downloading that exact artifact, compare its bytes with the source-reported expected digest:

printf '%s  %s\n' '3c3f392886ecad521467d5a01785d256853d5aaaf08dd3ef3fa887a364e36d32' 'python3-tokenizers-0.22.2-2.fc43.aarch64.rpm' | sha256sum --check --strict -

A match establishes equality with the repository metadata value. It does not establish safety or catalog-side artifact retrieval.

Field source: Fedora 43 updates aarch64 revision 43-updates-aarch64:ac6921c749da1e864e0971122151b63bb17e427965b3d9f9312a07b49ff10cac

0.22.2-2.fc43 / x86_64Observed Sep 1, 2026 to Sep 1, 2026

Verification status: Metadata observed; artifact bytes were not independently fetched or hashed by this catalog import. The digest below is source-reported.

Source-reported sha256: 15d077ac7388ab507dd9b2c649d0c157a015037c627b2bf9e8fbecbae1da0b41

After downloading that exact artifact, compare its bytes with the source-reported expected digest:

printf '%s  %s\n' '15d077ac7388ab507dd9b2c649d0c157a015037c627b2bf9e8fbecbae1da0b41' 'python3-tokenizers-0.22.2-2.fc43.x86_64.rpm' | sha256sum --check --strict -

A match establishes equality with the repository metadata value. It does not establish safety or catalog-side artifact retrieval.

Field source: Fedora 43 updates x86_64 revision 43-updates-x86_64:d59481a1758810a842f6aaef03eff437442211002d934e2d10266183c629137a

Complétude de la fiche

The completeness score measures metadata coverage, not software quality, security, compatibility, or suitability.

Summary and description: 25/25
Artifact path and source digest: 25/25
Dependency metadata: 15/15
Package-file index: 15/15
Homepage: 5/5
License text: 5/5
Source package or maintainer: 10/10

Recorded total: 100/100

Field source: Fedora 43 updates aarch64 revision 43-updates-aarch64:ac6921c749da1e864e0971122151b63bb17e427965b3d9f9312a07b49ff10cac, Fedora 43 updates x86_64 revision 43-updates-x86_64:d59481a1758810a842f6aaef03eff437442211002d934e2d10266183c629137a. The cross-OS mapping is catalog-derived from the source-reported homepage; it does not establish authorship or publisher identity

Sources et provenance

Field-source links above resolve here. Each source entry names the metadata publisher, trust tier, exact snapshot revision, signature result, and observation time; catalog-derived mappings are labeled separately.

Fedora 43 updates aarch64Sep 1, 2026
Authoritative source; repository metadata signature verified, revision 43-updates-aarch64:ac6921c749da1e864e0971122151b63bb17e427965b3d9f9312a07b49ff10cac
Signature verification covers the configured repository metadata chain. It does not certify that the package is safe or suitable.
Repository-signature verification record
Signed object
https://mirrors.fedoraproject.org/metalink?repo=updates-released-f43&arch=aarch64
Signed-object SHA-256
0af1807dead0a1d28599a942743e244fe57da21cf6a935b3e4155f4a3f1070a1
Signer fingerprint
Not recorded
Keyring revision
RPM-GPG-KEY-fedora-43-primary
SHA-256 2b1449a082d3264dda8e18369f04e9ac4163bf3f8cb530b0783dc2ab064a08ec
Tool and policy
1
openfactory-software-catalog-signature-v1
Verification time
Sep 1, 2026
Signed Release → package-index hash linkage
Path: Not recorded
Expected SHA-256: Not recorded
Observed SHA-256: Not recorded
Result: match verified
Fedora 43 updates x86_64Sep 1, 2026
Authoritative source; repository metadata signature verified, revision 43-updates-x86_64:d59481a1758810a842f6aaef03eff437442211002d934e2d10266183c629137a
Signature verification covers the configured repository metadata chain. It does not certify that the package is safe or suitable.
Repository-signature verification record
Signed object
https://mirrors.fedoraproject.org/metalink?repo=updates-released-f43&arch=x86_64
Signed-object SHA-256
3ae5fecc3db8c7b4ce0cf54a8ea572ed4694e7e132e201c2f6ab6570259d1061
Signer fingerprint
Not recorded
Keyring revision
RPM-GPG-KEY-fedora-43-primary
SHA-256 2b1449a082d3264dda8e18369f04e9ac4163bf3f8cb530b0783dc2ab064a08ec
Tool and policy
1
openfactory-software-catalog-signature-v1
Verification time
Sep 1, 2026
Signed Release → package-index hash linkage
Path: Not recorded
Expected SHA-256: Not recorded
Observed SHA-256: Not recorded
Result: match verified