openSUSE Tumbleweed native package

python313-tokenizers

Provides an implementation of today's most used tokenizers

Packages / openSUSE Tumbleweed / Unspecified / python313-tokenizers

[Source: python-tokenizers]

Package: python313-tokenizers (0.23.1-1.2)

External Resources:

Homepage: [github.com]

Similar packages:

[python314-tokenizers]
Provides an implementation of today's most used tokenizers

Provides an implementation of today's most used tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. * Train new vocabularies and tokenize, using today's most used tokenizers. * Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU. * Easy to use, but also extremely versatile. * Designed for research and production. * Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token. * Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Other Packages Related to python313-tokenizers:

dep: ld-linux-x86-64.so.2()(64bit)
Package not available
dep: ld-linux-x86-64.so.2(GLIBC_2.3)(64bit)
Package not available
dep: libgcc_s.so.1()(64bit)
Package not available
dep: libgcc_s.so.1(GCC_3.0)(64bit)
Package not available
dep: libgcc_s.so.1(GCC_3.3)(64bit)
Package not available
dep: libgcc_s.so.1(GCC_4.2.0)(64bit)
Package not available
dep: libm.so.6()(64bit)
Package not available
dep: libm.so.6(GLIBC_2.29)(64bit)
Package not available
dep: libstdc++.so.6()(64bit)
Package not available
dep: libstdc++.so.6(CXXABI_1.3)(64bit)
Package not available
dep: libstdc++.so.6(CXXABI_1.3.8)(64bit)
Package not available
dep: libstdc++.so.6(GLIBCXX_3.4)(64bit)
Package not available
dep: python(abi) (= 3.13)
Package not available
dep: [python313-huggingface-hub]
Client library for interaction with the huggingface hub
dep: libc.so.6(GLIBC_2.34)(64bit)
Package not available

Download python313-tokenizers

Architecture	Package Size	Installed Size	Files
x86_64	2.0 MiB	7.6 MiB	[list of files]

Package file paths (88)

Paths come from the repository package-file index for the observed builds. They describe archive/package associations, not every file that will exist on a running system after maintainer scripts, alternatives, generated state, diversions, or installation choices.

/usr/lib64/python3.13/site-packages/tokenizers
/usr/lib64/python3.13/site-packages/tokenizers-0.23.1.dist-info
/usr/lib64/python3.13/site-packages/tokenizers-0.23.1.dist-info/INSTALLER
/usr/lib64/python3.13/site-packages/tokenizers-0.23.1.dist-info/METADATA
/usr/lib64/python3.13/site-packages/tokenizers-0.23.1.dist-info/RECORD
/usr/lib64/python3.13/site-packages/tokenizers-0.23.1.dist-info/REQUESTED
/usr/lib64/python3.13/site-packages/tokenizers-0.23.1.dist-info/WHEEL
/usr/lib64/python3.13/site-packages/tokenizers/decoders
/usr/lib64/python3.13/site-packages/tokenizers/decoders/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/decoders/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/decoders.pyi
/usr/lib64/python3.13/site-packages/tokenizers/implementations
/usr/lib64/python3.13/site-packages/tokenizers/implementations/base_tokenizer.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/bert_wordpiece.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/byte_level_bpe.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/char_level_bpe.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/sentencepiece_bpe.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/sentencepiece_unigram.py
/usr/lib64/python3.13/site-packages/tokenizers/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/__init__.pyi
/usr/lib64/python3.13/site-packages/tokenizers/models
/usr/lib64/python3.13/site-packages/tokenizers/models/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/models/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/models/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/models/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/models.pyi
/usr/lib64/python3.13/site-packages/tokenizers/normalizers
/usr/lib64/python3.13/site-packages/tokenizers/normalizers/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/normalizers/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/normalizers.pyi
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers.pyi
/usr/lib64/python3.13/site-packages/tokenizers/processors
/usr/lib64/python3.13/site-packages/tokenizers/processors/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/processors/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/processors/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/processors/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/processors.pyi
/usr/lib64/python3.13/site-packages/tokenizers/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/py.typed
/usr/lib64/python3.13/site-packages/tokenizers/tokenizers.abi3.so
/usr/lib64/python3.13/site-packages/tokenizers/tokenizers.pyi
/usr/lib64/python3.13/site-packages/tokenizers/tools
/usr/lib64/python3.13/site-packages/tokenizers/tools/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/tools/visualizer.py
/usr/lib64/python3.13/site-packages/tokenizers/tools/visualizer-styles.css
/usr/lib64/python3.13/site-packages/tokenizers/trainers
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__init__.pyi
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/trainers.pyi
/usr/share/doc/packages/python313-tokenizers
/usr/share/doc/packages/python313-tokenizers/README.md
/usr/share/licenses/python313-tokenizers
/usr/share/licenses/python313-tokenizers/LICENSE

Field source: openSUSE Tumbleweed OSS revision tumbleweed-oss-multiarch:4db10cdde3ad82c7e22be35753c9e164b4bdd30f92db4aee17a0d4c94c5f902f

Use this package

OpenFactory can boot this operating system in a browser VM, or start a build that includes the native package name from this record.

Try openSUSE Tumbleweed in the browser Build an image that includes python313-tokenizers

Versions, suites, and repositories

Each row is recorded package-index metadata for one version, architecture, suite, and repository. Names, URLs, and sizes are source-reported; a link is a potentially mutable retrieval location, not an OpenFactory redistribution claim or proof that OpenFactory retained the artifact bytes.

Version	Release	Architecture	Repository	Package size	Installed size	Publisher repository artifact
0.23.1-1.2	tumbleweed / oss	x86_64	openSUSE Tumbleweed · OSS · multi-architecture	2.0 MiB	7.6 MiB	x86_64/python313-tokenizers-0.23.1-1.2.x86_64.rpm

Field source: openSUSE Tumbleweed OSS revision tumbleweed-oss-multiarch:4db10cdde3ad82c7e22be35753c9e164b4bdd30f92db4aee17a0d4c94c5f902f

Checksums and observation dates

For an APT source, signature verification authenticates the repository metadata chain and the Packages index containing this source-reported artifact digest. It does not certify package safety.

0.23.1-1.2 / x86_64Observed Sep 1, 2026 to Sep 1, 2026

Verification status: Metadata observed; artifact bytes were not independently fetched or hashed by this catalog import. The digest below is source-reported.

Source-reported sha512: 3b4c0976075d281d8cfd4d966bd5e83c6e8bbf68aaa92c756ba940493172a2b2c58068b0fe94bbae37abbc79c39b17389607ff4561295ff65c4d54ccc761ab87

Field source: openSUSE Tumbleweed OSS revision tumbleweed-oss-multiarch:4db10cdde3ad82c7e22be35753c9e164b4bdd30f92db4aee17a0d4c94c5f902f

Catalog record completeness

The completeness score measures metadata coverage, not software quality, security, compatibility, or suitability.

Summary and description: 25/25
Artifact path and source digest: 25/25
Dependency metadata: 15/15
Package-file index: 15/15
Homepage: 5/5
License text: 5/5
Source package or maintainer: 10/10

Recorded total: 100/100

Field source: openSUSE Tumbleweed OSS revision tumbleweed-oss-multiarch:4db10cdde3ad82c7e22be35753c9e164b4bdd30f92db4aee17a0d4c94c5f902f. The cross-OS mapping is catalog-derived from the source-reported homepage; it does not establish authorship or publisher identity

Sources and provenance

Field-source links above resolve here. Each source entry names the metadata publisher, trust tier, exact snapshot revision, signature result, and observation time; catalog-derived mappings are labeled separately.

openSUSE Tumbleweed OSSSep 1, 2026
Authoritative source; repository metadata signature verified, revision tumbleweed-oss-multiarch:4db10cdde3ad82c7e22be35753c9e164b4bdd30f92db4aee17a0d4c94c5f902f
Signature verification covers the configured repository metadata chain. It does not certify that the package is safe or suitable.
Repository-signature verification record
Signed object
https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml
Signed-object SHA-256
4db10cdde3ad82c7e22be35753c9e164b4bdd30f92db4aee17a0d4c94c5f902f
Signer fingerprint
AD485664E901B867051AB15F35A2F86E29B700A4
Keyring revision
opensuse-project-signing-key.asc
SHA-256 b5745739ebfb95b25b8e810f9bcb847fe750ccda598bd85f02f0e974599a6d7e
Tool and policy
gpgv (GnuPG) 2.4.9
openfactory-software-catalog-signature-v1
Verification time
Sep 1, 2026
Signed Release → package-index hash linkage
Path: Not recorded
Expected SHA-256: Not recorded
Observed SHA-256: Not recorded
Result: match verified