openSUSE Leap 16.0 native package

python313-tokenizers

Proporciona una implementación de los tokenizadores más usados hoy en día

Packages / openSUSE Leap 16.0 / Unspecified / python313-tokenizers

[Source: python-tokenizers]

Package: python313-tokenizers (0.21.1-bp160.1.11)

[Project overview: python-tokenizers]

External Resources:

Homepage: [github.com]

Proporciona una implementación de los tokenizadores más usados hoy en día

Proporciona una implementación de los tokenizadores más usados hoy en día, con énfasis en el rendimiento y la versatilidad. * Entrena nuevos vocabularios y tokeniza, usando los tokenizadores más usados hoy en día. * Extremadamente rápido (tanto el entrenamiento como la tokenización), gracias a la implementación en Rust. Tarda menos de 20 segundos en tokenizar un GB de texto en la CPU de un servidor. * Fácil de usar, pero también extremadamente versátil. * Diseñado para investigación y producción. * La normalización incluye seguimiento de alineaciones. Siempre es posible obtener la parte de la oración original que corresponde a un token dado. * Realiza todo el preprocesamiento: Truncate, Pad, añadir los tokens especiales que tu modelo necesita.

Other Packages Related to python313-tokenizers:

dep: libgcc_s.so.1()(64bit)
Package not available
dep: libgcc_s.so.1(GCC_3.0)(64bit)
Package not available
dep: libgcc_s.so.1(GCC_3.3)(64bit)
Package not available
dep: libgcc_s.so.1(GCC_4.2.0)(64bit)
Package not available
dep: libm.so.6()(64bit)
Package not available
dep: libm.so.6(GLIBC_2.29)(64bit)
Package not available
dep: libstdc++.so.6()(64bit)
Package not available
dep: libstdc++.so.6(CXXABI_1.3)(64bit)
Package not available
dep: libstdc++.so.6(CXXABI_1.3.8)(64bit)
Package not available
dep: libstdc++.so.6(GLIBCXX_3.4)(64bit)
Package not available
dep: python(abi) (= 3.13)
Package not available
dep: [python313-huggingface-hub]
Biblioteca cliente para interactuar con el hub de huggingface
dep: libc.so.6(GLIBC_2.34)(64bit)
Package not available

Download python313-tokenizers

Architecture	Package Size	Installed Size	Files
aarch64	1.8 MiB	6.4 MiB	[list of files]
x86_64	1.9 MiB	7.1 MiB	[list of files]

Rutas de archivos del paquete (85)

Paths come from the repository package-file index for the observed builds. They describe archive/package associations, not every file that will exist on a running system after maintainer scripts, alternatives, generated state, diversions, or installation choices.

/usr/lib64/python3.13/site-packages/tokenizers
/usr/lib64/python3.13/site-packages/tokenizers-0.21.1.dist-info
/usr/lib64/python3.13/site-packages/tokenizers-0.21.1.dist-info/INSTALLER
/usr/lib64/python3.13/site-packages/tokenizers-0.21.1.dist-info/METADATA
/usr/lib64/python3.13/site-packages/tokenizers-0.21.1.dist-info/RECORD
/usr/lib64/python3.13/site-packages/tokenizers-0.21.1.dist-info/REQUESTED
/usr/lib64/python3.13/site-packages/tokenizers-0.21.1.dist-info/WHEEL
/usr/lib64/python3.13/site-packages/tokenizers/decoders
/usr/lib64/python3.13/site-packages/tokenizers/decoders/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/decoders/__init__.pyi
/usr/lib64/python3.13/site-packages/tokenizers/decoders/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations
/usr/lib64/python3.13/site-packages/tokenizers/implementations/base_tokenizer.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/bert_wordpiece.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/byte_level_bpe.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/char_level_bpe.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/implementations/sentencepiece_bpe.py
/usr/lib64/python3.13/site-packages/tokenizers/implementations/sentencepiece_unigram.py
/usr/lib64/python3.13/site-packages/tokenizers/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/__init__.pyi
/usr/lib64/python3.13/site-packages/tokenizers/models
/usr/lib64/python3.13/site-packages/tokenizers/models/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/models/__init__.pyi
/usr/lib64/python3.13/site-packages/tokenizers/models/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/models/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/models/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/normalizers
/usr/lib64/python3.13/site-packages/tokenizers/normalizers/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/normalizers/__init__.pyi
/usr/lib64/python3.13/site-packages/tokenizers/normalizers/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers/__init__.pyi
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/processors
/usr/lib64/python3.13/site-packages/tokenizers/processors/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/processors/__init__.pyi
/usr/lib64/python3.13/site-packages/tokenizers/processors/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/processors/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/processors/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/tokenizers.abi3.so
/usr/lib64/python3.13/site-packages/tokenizers/tools
/usr/lib64/python3.13/site-packages/tokenizers/tools/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__/__init__.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-313.pyc
/usr/lib64/python3.13/site-packages/tokenizers/tools/visualizer.py
/usr/lib64/python3.13/site-packages/tokenizers/tools/visualizer-styles.css
/usr/lib64/python3.13/site-packages/tokenizers/trainers
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__init__.py
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__init__.pyi
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__pycache__
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-313.opt-1.pyc
/usr/lib64/python3.13/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-313.pyc
/usr/share/doc/packages/python313-tokenizers
/usr/share/doc/packages/python313-tokenizers/README.md
/usr/share/licenses/python313-tokenizers
/usr/share/licenses/python313-tokenizers/LICENSE

Field source: openSUSE Leap 16.0 OSS revision leap-16-0-oss-multiarch:271b36d440253edf9871ac0c100319fc7443d8f2d54b6b449a72e9414c51577c

Usar este paquete

OpenFactory puede arrancar este sistema operativo en una máquina virtual del navegador, o iniciar una construcción que incluye el nombre nativo del paquete de este registro.

Probar openSUSE Leap 16.0 en el navegador Construir una imagen que incluye python313-tokenizers

Versiones, suites y repositorios

Cada fila es metadato del índice de paquetes para una versión, arquitectura, suite y repositorio. Nombres, URL y tamaños vienen de la fuente; un enlace es un punto de descarga que puede cambiar, no una redistribución de OpenFactory.

Version	Release	Architecture	Repository	Package size	Installed size	Publisher repository artifact
0.21.1-bp160.1.11	leap-16.0 / oss	aarch64	openSUSE Leap 16.0 · OSS · multi-architecture	1.8 MiB	6.4 MiB	aarch64/python313-tokenizers-0.21.1-bp160.1.11.aarch64.rpm
0.21.1-bp160.1.11	leap-16.0 / oss	x86_64	openSUSE Leap 16.0 · OSS · multi-architecture	1.9 MiB	7.1 MiB	x86_64/python313-tokenizers-0.21.1-bp160.1.11.x86_64.rpm

Field source: openSUSE Leap 16.0 OSS revision leap-16-0-oss-multiarch:271b36d440253edf9871ac0c100319fc7443d8f2d54b6b449a72e9414c51577c

Sumas de comprobación y fechas de observación

For an APT source, signature verification authenticates the repository metadata chain and the Packages index containing this source-reported artifact digest. It does not certify package safety.

0.21.1-bp160.1.11 / aarch64Observed Sep 1, 2026 to Sep 1, 2026

Verification status: Metadata observed; artifact bytes were not independently fetched or hashed by this catalog import. The digest below is source-reported.

Source-reported sha512: 57fd6c0f7794a13f229969aed58484cf588e8db5f4737297b3465326f9db2de9b6bf03abdc384140507887149c42fc9d566f045448c8513ad5067be3b885f617

Field source: openSUSE Leap 16.0 OSS revision leap-16-0-oss-multiarch:271b36d440253edf9871ac0c100319fc7443d8f2d54b6b449a72e9414c51577c

0.21.1-bp160.1.11 / x86_64Observed Sep 1, 2026 to Sep 1, 2026

Verification status: Metadata observed; artifact bytes were not independently fetched or hashed by this catalog import. The digest below is source-reported.

Source-reported sha512: 84c41244f188f4054026bb637d89a51a8f92b6d51a43f0d7e90e3c908d1ba4add00a7d9bd1eb9019a57e300abe74dcd29d8d68953384572d3e110ec5884c21c7

Field source: openSUSE Leap 16.0 OSS revision leap-16-0-oss-multiarch:271b36d440253edf9871ac0c100319fc7443d8f2d54b6b449a72e9414c51577c

Completitud del registro

The completeness score measures metadata coverage, not software quality, security, compatibility, or suitability.

Summary and description: 25/25
Artifact path and source digest: 25/25
Dependency metadata: 15/15
Package-file index: 15/15
Homepage: 5/5
License text: 5/5
Source package or maintainer: 10/10

Recorded total: 100/100

Field source: openSUSE Leap 16.0 OSS revision leap-16-0-oss-multiarch:271b36d440253edf9871ac0c100319fc7443d8f2d54b6b449a72e9414c51577c. The cross-OS mapping is catalog-derived from the source-reported homepage; it does not establish authorship or publisher identity

Fuentes y procedencia

Field-source links above resolve here. Each source entry names the metadata publisher, trust tier, exact snapshot revision, signature result, and observation time; catalog-derived mappings are labeled separately.

openSUSE Leap 16.0 OSSSep 1, 2026
Authoritative source; repository metadata signature verified, revision leap-16-0-oss-multiarch:271b36d440253edf9871ac0c100319fc7443d8f2d54b6b449a72e9414c51577c
Signature verification covers the configured repository metadata chain. It does not certify that the package is safe or suitable.
Repository-signature verification record
Signed object
https://download.opensuse.org/distribution/leap/16.0/repo/oss/repodata/repomd.xml
Signed-object SHA-256
271b36d440253edf9871ac0c100319fc7443d8f2d54b6b449a72e9414c51577c
Signer fingerprint
AD485664E901B867051AB15F35A2F86E29B700A4
Keyring revision
opensuse-project-signing-key.asc
SHA-256 b5745739ebfb95b25b8e810f9bcb847fe750ccda598bd85f02f0e974599a6d7e
Tool and policy
gpgv (GnuPG) 2.4.9
openfactory-software-catalog-signature-v1
Verification time
Sep 1, 2026
Signed Release → package-index hash linkage
Path: Not recorded
Expected SHA-256: Not recorded
Observed SHA-256: Not recorded
Result: match verified