Fedora 44 native package

python3-tokenizers

Implementation of today's most used tokenizers

Packages / Fedora 44 / Unspecified / python3-tokenizers

[Source: python-tokenizers]

Package: python3-tokenizers (0.22.2-2.fc44)

[Project overview: python-tokenizers]

External Resources:

Homepage: [github.com]

Implementation of today's most used tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Bindings over the rust-tokenizers implementation.

Other Packages Related to python3-tokenizers:

dep: (python3.14dist(huggingface-hub) < 2~~ with python3.14dist(huggingface-hub) >= 0.16.4)
Package not available
dep: ld-linux-aarch64.so.1()(64bit)
Package not available
dep: ld-linux-aarch64.so.1(GLIBC_2.17)(64bit)
Package not available
dep: libgcc_s.so.1()(64bit)
Package not available
dep: libgcc_s.so.1(GCC_3.0)(64bit)
Package not available
dep: libgcc_s.so.1(GCC_3.3)(64bit)
Package not available
dep: libgcc_s.so.1(GCC_4.2.0)(64bit)
Package not available
dep: libm.so.6()(64bit)
Package not available
dep: libm.so.6(GLIBC_2.29)(64bit)
Package not available
dep: libonig.so.5()(64bit)
Package not available
dep: libstdc++.so.6()(64bit)
Package not available
dep: libstdc++.so.6(CXXABI_1.3)(64bit)
Package not available
dep: libstdc++.so.6(GLIBCXX_3.4)(64bit)
Package not available
dep: libstdc++.so.6(GLIBCXX_3.4.30)(64bit)
Package not available
dep: python(abi) (= 3.14)
Package not available
dep: rtld(GNU_HASH)
Package not available
dep: libc.so.6(GLIBC_2.34)(64bit)
Package not available

Download python3-tokenizers

Architecture	Package Size	Installed Size	Files
aarch64	1.8 MiB	5.7 MiB	[list of files]
x86_64	2.0 MiB	6.1 MiB	[list of files]

Paketdateipfade (98)

Paths come from the repository package-file index for the observed builds. They describe archive/package associations, not every file that will exist on a running system after maintainer scripts, alternatives, generated state, diversions, or installation choices.

/usr/lib64/python3.14/site-packages/tokenizers
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/INSTALLER
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/licenses
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/licenses/LICENSE
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/licenses/LICENSE.dependencies
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/METADATA
/usr/lib64/python3.14/site-packages/tokenizers-0.22.2.dist-info/WHEEL
/usr/lib64/python3.14/site-packages/tokenizers/decoders
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations
/usr/lib64/python3.14/site-packages/tokenizers/implementations/base_tokenizer.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/bert_wordpiece.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/byte_level_bpe.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/char_level_bpe.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/implementations/sentencepiece_bpe.py
/usr/lib64/python3.14/site-packages/tokenizers/implementations/sentencepiece_unigram.py
/usr/lib64/python3.14/site-packages/tokenizers/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/models
/usr/lib64/python3.14/site-packages/tokenizers/models/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/models/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/models/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/models/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/models/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/normalizers
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/processors
/usr/lib64/python3.14/site-packages/tokenizers/processors/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/processors/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/processors/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/processors/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/processors/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tokenizers.abi3.so
/usr/lib64/python3.14/site-packages/tokenizers/tokenizers.pyi
/usr/lib64/python3.14/site-packages/tokenizers/tools
/usr/lib64/python3.14/site-packages/tokenizers/tools/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/__init__.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-314.pyc
/usr/lib64/python3.14/site-packages/tokenizers/tools/visualizer.py
/usr/lib64/python3.14/site-packages/tokenizers/tools/visualizer-styles.css
/usr/lib64/python3.14/site-packages/tokenizers/trainers
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__init__.py
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__init__.pyi
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__pycache__
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-314.opt-1.pyc
/usr/lib64/python3.14/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-314.pyc
/usr/lib/.build-id
/usr/lib/.build-id/28
/usr/lib/.build-id/28/900e28649597b4aba94de917e0f6c2b40912fc
/usr/lib/.build-id/b1
/usr/lib/.build-id/b1/b52b114ee9210d855a55a4f0013bda064eaad0
/usr/lib/.build-id/cf
/usr/lib/.build-id/cf/042ea6b99d0adde37eec0b5980f03f5d0bb7c4
/usr/lib/.build-id/d4
/usr/lib/.build-id/d4/fa252e2404762e2581a5eea8aa606af48ae22a
/usr/share/doc/python3-tokenizers
/usr/share/doc/python3-tokenizers/CHANGELOG.md
/usr/share/doc/python3-tokenizers/README.md
/usr/share/licenses/python3-tokenizers
/usr/share/licenses/python3-tokenizers/LICENSE
/usr/share/licenses/python3-tokenizers/LICENSE.dependencies

Field source: Fedora 44 updates aarch64 revision 44-updates-aarch64:6ecf7f9a3a707e537dd0c542e2f5fabb365c3c76c03b091b23e9c82d7f8ff88b

Dieses Paket verwenden

OpenFactory kann dieses Betriebssystem in einer Browser-VM starten oder einen Image-Build mit dem nativen Paketnamen aus diesem Datensatz beginnen.

Fedora 44 im Browser testen Image mit python3-tokenizers bauen

Versionen, Suiten und Repositories

Jede Zeile ist Paketindex-Metadaten für eine Version, Architektur, Suite und ein Repository. Namen, URLs und Größen stammen aus der Quelle; ein Link ist ein veränderbarer Abrufort, kein Weitergabanspruch von OpenFactory.

Version	Release	Architecture	Repository	Package size	Installed size	Publisher repository artifact
0.22.2-2.fc44	44 / updates	aarch64	Fedora 44 · updates · aarch64	1.8 MiB	5.7 MiB	Packages/p/python3-tokenizers-0.22.2-2.fc44.aarch64.rpm
0.22.2-2.fc44	44 / updates	x86_64	Fedora 44 · updates · x86_64	2.0 MiB	6.1 MiB	Packages/p/python3-tokenizers-0.22.2-2.fc44.x86_64.rpm
0.22.2-1.fc44	44 / everything	aarch64	Fedora 44 · Everything · aarch64	1.8 MiB	5.7 MiB	Packages/p/python3-tokenizers-0.22.2-1.fc44.aarch64.rpm
0.22.2-1.fc44	44 / everything	x86_64	Fedora 44 · Everything · x86_64	2.0 MiB	6.1 MiB	Packages/p/python3-tokenizers-0.22.2-1.fc44.x86_64.rpm

Field source: Fedora 44 updates aarch64 revision 44-updates-aarch64:6ecf7f9a3a707e537dd0c542e2f5fabb365c3c76c03b091b23e9c82d7f8ff88b

Prüfsummen und Beobachtungsdaten

For an APT source, signature verification authenticates the repository metadata chain and the Packages index containing this source-reported artifact digest. It does not certify package safety.

0.22.2-2.fc44 / aarch64Observed Sep 1, 2026 to Sep 1, 2026

Verification status: Metadata observed; artifact bytes were not independently fetched or hashed by this catalog import. The digest below is source-reported.

Source-reported sha256: 5260ab59267944be0b22958fbc4630a3e32d448b91da9039fbc1d4de6039b6e7

After downloading that exact artifact, compare its bytes with the source-reported expected digest:

printf '%s  %s\n' '5260ab59267944be0b22958fbc4630a3e32d448b91da9039fbc1d4de6039b6e7' 'python3-tokenizers-0.22.2-2.fc44.aarch64.rpm' | sha256sum --check --strict -

A match establishes equality with the repository metadata value. It does not establish safety or catalog-side artifact retrieval.

Field source: Fedora 44 updates aarch64 revision 44-updates-aarch64:6ecf7f9a3a707e537dd0c542e2f5fabb365c3c76c03b091b23e9c82d7f8ff88b

0.22.2-2.fc44 / x86_64Observed Sep 1, 2026 to Sep 1, 2026

Verification status: Metadata observed; artifact bytes were not independently fetched or hashed by this catalog import. The digest below is source-reported.

Source-reported sha256: 4c04d391199efd0e1ce8424d6ac40eea73cc9bb47a7a0b197d61dd39c683b4e9

After downloading that exact artifact, compare its bytes with the source-reported expected digest:

printf '%s  %s\n' '4c04d391199efd0e1ce8424d6ac40eea73cc9bb47a7a0b197d61dd39c683b4e9' 'python3-tokenizers-0.22.2-2.fc44.x86_64.rpm' | sha256sum --check --strict -

A match establishes equality with the repository metadata value. It does not establish safety or catalog-side artifact retrieval.

Field source: Fedora 44 updates x86_64 revision 44-updates-x86_64:ffcac72d41afd8a62019714dd668a27f333f8ac19678580178b5b67c23ca6419

0.22.2-1.fc44 / aarch64Observed Sep 1, 2026 to Sep 1, 2026

Verification status: Metadata observed; artifact bytes were not independently fetched or hashed by this catalog import. The digest below is source-reported.

Source-reported sha256: ac24dd932041c4dc5d264f4f20d637db485c9cbfb04811ac02bb71d2d606add3

After downloading that exact artifact, compare its bytes with the source-reported expected digest:

printf '%s  %s\n' 'ac24dd932041c4dc5d264f4f20d637db485c9cbfb04811ac02bb71d2d606add3' 'python3-tokenizers-0.22.2-1.fc44.aarch64.rpm' | sha256sum --check --strict -

A match establishes equality with the repository metadata value. It does not establish safety or catalog-side artifact retrieval.

Field source: Fedora 44 Everything aarch64 revision 44-everything-aarch64:ad124f8125666e9059d7a8180427bdaea80f6286f71140457e5c7edd95883eee

0.22.2-1.fc44 / x86_64Observed Sep 1, 2026 to Sep 1, 2026

Verification status: Metadata observed; artifact bytes were not independently fetched or hashed by this catalog import. The digest below is source-reported.

Source-reported sha256: bd43b8a0e3eaa8a7831fc29b3acdedcce267956cd32d42a128cc23ac9215ed7b

After downloading that exact artifact, compare its bytes with the source-reported expected digest:

printf '%s  %s\n' 'bd43b8a0e3eaa8a7831fc29b3acdedcce267956cd32d42a128cc23ac9215ed7b' 'python3-tokenizers-0.22.2-1.fc44.x86_64.rpm' | sha256sum --check --strict -

A match establishes equality with the repository metadata value. It does not establish safety or catalog-side artifact retrieval.

Field source: Fedora 44 Everything x86_64 revision 44-everything-x86_64:da3845427d188097f6fd71b417a039bdfb8efefc4f38ca44b5cbb94f95a18991

Vollständigkeit des Katalogsatzes

The completeness score measures metadata coverage, not software quality, security, compatibility, or suitability.

Summary and description: 25/25
Artifact path and source digest: 25/25
Dependency metadata: 15/15
Package-file index: 15/15
Homepage: 5/5
License text: 5/5
Source package or maintainer: 10/10

Recorded total: 100/100

Field source: Fedora 44 Everything aarch64 revision 44-everything-aarch64:ad124f8125666e9059d7a8180427bdaea80f6286f71140457e5c7edd95883eee, Fedora 44 Everything x86_64 revision 44-everything-x86_64:da3845427d188097f6fd71b417a039bdfb8efefc4f38ca44b5cbb94f95a18991, Fedora 44 updates aarch64 revision 44-updates-aarch64:6ecf7f9a3a707e537dd0c542e2f5fabb365c3c76c03b091b23e9c82d7f8ff88b, Fedora 44 updates x86_64 revision 44-updates-x86_64:ffcac72d41afd8a62019714dd668a27f333f8ac19678580178b5b67c23ca6419. The cross-OS mapping is catalog-derived from the source-reported homepage; it does not establish authorship or publisher identity

Quellen und Herkunft

Field-source links above resolve here. Each source entry names the metadata publisher, trust tier, exact snapshot revision, signature result, and observation time; catalog-derived mappings are labeled separately.

Fedora 44 Everything aarch64Sep 1, 2026
Authoritative source; repository metadata signature verified, revision 44-everything-aarch64:ad124f8125666e9059d7a8180427bdaea80f6286f71140457e5c7edd95883eee
Signature verification covers the configured repository metadata chain. It does not certify that the package is safe or suitable.
Repository-signature verification record
Signed object
https://mirrors.fedoraproject.org/metalink?repo=fedora-44&arch=aarch64
Signed-object SHA-256
236514c32119ab3c85b81fbb9e5e9d92b07aeba0d19d64c9b4fc9117a1863769
Signer fingerprint
Not recorded
Keyring revision
RPM-GPG-KEY-fedora-44-primary
SHA-256 93642aec521a1e5e96dd715f7ae0ec0850ebc9de09a94ce03cae5263f26cc18a
Tool and policy
1
openfactory-software-catalog-signature-v1
Verification time
Sep 1, 2026
Signed Release → package-index hash linkage
Path: Not recorded
Expected SHA-256: Not recorded
Observed SHA-256: Not recorded
Result: match verified
Fedora 44 Everything x86_64Sep 1, 2026
Authoritative source; repository metadata signature verified, revision 44-everything-x86_64:da3845427d188097f6fd71b417a039bdfb8efefc4f38ca44b5cbb94f95a18991
Signature verification covers the configured repository metadata chain. It does not certify that the package is safe or suitable.
Repository-signature verification record
Signed object
https://mirrors.fedoraproject.org/metalink?repo=fedora-44&arch=x86_64
Signed-object SHA-256
1300b99ac5d04b5561ad901320c8bd59ed1553f4971c3ce2ca48c40242c66676
Signer fingerprint
Not recorded
Keyring revision
RPM-GPG-KEY-fedora-44-primary
SHA-256 93642aec521a1e5e96dd715f7ae0ec0850ebc9de09a94ce03cae5263f26cc18a
Tool and policy
1
openfactory-software-catalog-signature-v1
Verification time
Sep 1, 2026
Signed Release → package-index hash linkage
Path: Not recorded
Expected SHA-256: Not recorded
Observed SHA-256: Not recorded
Result: match verified
Fedora 44 updates aarch64Sep 1, 2026
Authoritative source; repository metadata signature verified, revision 44-updates-aarch64:6ecf7f9a3a707e537dd0c542e2f5fabb365c3c76c03b091b23e9c82d7f8ff88b
Signature verification covers the configured repository metadata chain. It does not certify that the package is safe or suitable.
Repository-signature verification record
Signed object
https://mirrors.fedoraproject.org/metalink?repo=updates-released-f44&arch=aarch64
Signed-object SHA-256
2ec0ba92e556309075d8efe3c3301b36bb7ecd47c4e17566f0192c9c5436ed5f
Signer fingerprint
Not recorded
Keyring revision
RPM-GPG-KEY-fedora-44-primary
SHA-256 93642aec521a1e5e96dd715f7ae0ec0850ebc9de09a94ce03cae5263f26cc18a
Tool and policy
1
openfactory-software-catalog-signature-v1
Verification time
Sep 1, 2026
Signed Release → package-index hash linkage
Path: Not recorded
Expected SHA-256: Not recorded
Observed SHA-256: Not recorded
Result: match verified
Fedora 44 updates x86_64Sep 1, 2026
Authoritative source; repository metadata signature verified, revision 44-updates-x86_64:ffcac72d41afd8a62019714dd668a27f333f8ac19678580178b5b67c23ca6419
Signature verification covers the configured repository metadata chain. It does not certify that the package is safe or suitable.
Repository-signature verification record
Signed object
https://mirrors.fedoraproject.org/metalink?repo=updates-released-f44&arch=x86_64
Signed-object SHA-256
a69a0ab020457a7af2515066b7caccb3950bcf9dc7bedb52b1850098d9857ae1
Signer fingerprint
Not recorded
Keyring revision
RPM-GPG-KEY-fedora-44-primary
SHA-256 93642aec521a1e5e96dd715f7ae0ec0850ebc9de09a94ce03cae5263f26cc18a
Tool and policy
1
openfactory-software-catalog-signature-v1
Verification time
Sep 1, 2026
Signed Release → package-index hash linkage
Path: Not recorded
Expected SHA-256: Not recorded
Observed SHA-256: Not recorded
Result: match verified