admin/vLLM_Head_Run
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit of vLLM Head Run project files

Browse Source
This commit is contained in:
JK Woo
2026-06-26 13:57:21 +00:00
parent 35d659ac38
commit 5063aa98d8
1262 changed files with 245164 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
.venv/lib/python3.12/site-packages/81d243bd2c585b0f4821__mypyc.cpython-312-aarch64-linux-gnu.so Executable file
View File

Binary file not shown.

1
.venv/lib/python3.12/site-packages/certifi-2026.6.17.dist-info/INSTALLER Normal file
View File

@@ -0,0 +1 @@
pip

78
.venv/lib/python3.12/site-packages/certifi-2026.6.17.dist-info/METADATA Normal file
View File

@@ -0,0 +1,78 @@
Metadata-Version: 2.4
Name: certifi
Version: 2026.6.17
Summary: Python package for providing Mozilla's CA Bundle.
Home-page: https://github.com/certifi/python-certifi
Author: Kenneth Reitz
Author-email: me@kennethreitz.com
License: MPL-2.0
Project-URL: Source, https://github.com/certifi/python-certifi
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
Classifier: Natural Language :: English
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.7
License-File: LICENSE
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: requires-python
Dynamic: summary
Certifi: Python SSL Certificates
================================
Certifi provides Mozilla's carefully curated collection of Root Certificates for
validating the trustworthiness of SSL certificates while verifying the identity
of TLS hosts. It has been extracted from the `Requests`_ project.
Installation
------------
``certifi`` is available on PyPI. Simply install it with ``pip``::
$ pip install certifi
Usage
-----
To reference the installed certificate authority (CA) bundle, you can use the
built-in function::
>>> import certifi
>>> certifi.where()
'/usr/local/lib/python3.7/site-packages/certifi/cacert.pem'
Or from the command line::
$ python -m certifi
/usr/local/lib/python3.7/site-packages/certifi/cacert.pem
Enjoy!
.. _`Requests`: https://requests.readthedocs.io/en/latest/
Addition/Removal of Certificates
--------------------------------
Certifi does not support any addition/removal or other modification of the
CA trust store content. This project is intended to provide a reliable and
highly portable root of trust to python deployments. Look to upstream projects
for methods to use alternate trust.

14
.venv/lib/python3.12/site-packages/certifi-2026.6.17.dist-info/RECORD Normal file
View File

@@ -0,0 +1,14 @@
certifi-2026.6.17.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
certifi-2026.6.17.dist-info/METADATA,sha256=6hXAnt0a2el7xm2e9xvPuRCntZLjdKCkN81e47E0wN8,2474
certifi-2026.6.17.dist-info/RECORD,,
certifi-2026.6.17.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
certifi-2026.6.17.dist-info/licenses/LICENSE,sha256=6TcW2mucDVpKHfYP5pWzcPBpVgPSH2-D8FPkLPwQyvc,989
certifi-2026.6.17.dist-info/top_level.txt,sha256=KMu4vUCfsjLrkPbSNdgdekS-pVJzBAJFO__nI8NF6-U,8
certifi/__init__.py,sha256=-W1R_y8WCaSkT1tdjuxH_zTBZY1YH6xQgdN1nbBajOE,94
certifi/__main__.py,sha256=xBBoj905TUWBLRGANOcf7oi6e-3dMP4cEoG9OyMs11g,243
certifi/__pycache__/__init__.cpython-312.pyc,,
certifi/__pycache__/__main__.cpython-312.pyc,,
certifi/__pycache__/core.cpython-312.pyc,,
certifi/cacert.pem,sha256=u8fpwB11UbuKFZtd7dmJuO484QWv9SK2jrGwG_hUyrA,234354
certifi/core.py,sha256=XFXycndG5pf37ayeF8N32HUuDafsyhkVMbO4BAPWHa0,3394
certifi/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0

5
.venv/lib/python3.12/site-packages/certifi-2026.6.17.dist-info/WHEEL Normal file
View File

@@ -0,0 +1,5 @@
Wheel-Version: 1.0
Generator: setuptools (82.0.1)
Root-Is-Purelib: true
Tag: py3-none-any

20
.venv/lib/python3.12/site-packages/certifi-2026.6.17.dist-info/licenses/LICENSE Normal file
View File

@@ -0,0 +1,20 @@
This package contains a modified version of ca-bundle.crt:
ca-bundle.crt -- Bundle of CA Root Certificates
This is a bundle of X.509 certificates of public Certificate Authorities
(CA). These were automatically extracted from Mozilla's root certificates
file (certdata.txt). This file can be found in the mozilla source tree:
https://hg.mozilla.org/mozilla-central/file/tip/security/nss/lib/ckfw/builtins/certdata.txt
It contains the certificates in PEM format and therefore
can be directly used with curl / libcurl / php_curl, or with
an Apache+mod_ssl webserver for SSL client authentication.
Just configure this file as the SSLCACertificateFile.#
***** BEGIN LICENSE BLOCK *****
This Source Code Form is subject to the terms of the Mozilla Public License,
v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain
one at http://mozilla.org/MPL/2.0/.
***** END LICENSE BLOCK *****
@(#) $RCSfile: certdata.txt,v $ $Revision: 1.80 $ $Date: 2011/11/03 15:11:58 $

1
.venv/lib/python3.12/site-packages/certifi-2026.6.17.dist-info/top_level.txt Normal file
View File

@@ -0,0 +1 @@
certifi

4
.venv/lib/python3.12/site-packages/certifi/__init__.py Normal file
View File

@@ -0,0 +1,4 @@
from .core import contents, where
__all__ = ["contents", "where"]
__version__ = "2026.06.17"

12
.venv/lib/python3.12/site-packages/certifi/__main__.py Normal file
View File

@@ -0,0 +1,12 @@
import argparse
from certifi import contents, where
parser = argparse.ArgumentParser()
parser.add_argument("-c", "--contents", action="store_true")
args = parser.parse_args()
if args.contents:
print(contents())
else:
print(where())

BIN
.venv/lib/python3.12/site-packages/certifi/__pycache__/__init__.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/certifi/__pycache__/__main__.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/certifi/__pycache__/core.cpython-312.pyc Normal file
View File

Binary file not shown.

3863
.venv/lib/python3.12/site-packages/certifi/cacert.pem Normal file
View File

File diff suppressed because it is too large Load Diff

83
.venv/lib/python3.12/site-packages/certifi/core.py Normal file
View File

@@ -0,0 +1,83 @@
"""
certifi.py
~~~~~~~~~~
This module returns the installation location of cacert.pem or its contents.
"""
import sys
import atexit
def exit_cacert_ctx() -> None:
_CACERT_CTX.__exit__(None, None, None) # type: ignore[union-attr]
if sys.version_info >= (3, 11):
from importlib.resources import as_file, files
_CACERT_CTX = None
_CACERT_PATH = None
def where() -> str:
# This is slightly terrible, but we want to delay extracting the file
# in cases where we're inside of a zipimport situation until someone
# actually calls where(), but we don't want to re-extract the file
# on every call of where(), so we'll do it once then store it in a
# global variable.
global _CACERT_CTX
global _CACERT_PATH
if _CACERT_PATH is None:
# This is slightly janky, the importlib.resources API wants you to
# manage the cleanup of this file, so it doesn't actually return a
# path, it returns a context manager that will give you the path
# when you enter it and will do any cleanup when you leave it. In
# the common case of not needing a temporary file, it will just
# return the file system location and the __exit__() is a no-op.
#
# We also have to hold onto the actual context manager, because
# it will do the cleanup whenever it gets garbage collected, so
# we will also store that at the global level as well.
_CACERT_CTX = as_file(files("certifi").joinpath("cacert.pem"))
_CACERT_PATH = str(_CACERT_CTX.__enter__())
atexit.register(exit_cacert_ctx)
return _CACERT_PATH
def contents() -> str:
return files("certifi").joinpath("cacert.pem").read_text(encoding="ascii")
else:
from importlib.resources import path as get_path, read_text
_CACERT_CTX = None
_CACERT_PATH = None
def where() -> str:
# This is slightly terrible, but we want to delay extracting the
# file in cases where we're inside of a zipimport situation until
# someone actually calls where(), but we don't want to re-extract
# the file on every call of where(), so we'll do it once then store
# it in a global variable.
global _CACERT_CTX
global _CACERT_PATH
if _CACERT_PATH is None:
# This is slightly janky, the importlib.resources API wants you
# to manage the cleanup of this file, so it doesn't actually
# return a path, it returns a context manager that will give
# you the path when you enter it and will do any cleanup when
# you leave it. In the common case of not needing a temporary
# file, it will just return the file system location and the
# __exit__() is a no-op.
#
# We also have to hold onto the actual context manager, because
# it will do the cleanup whenever it gets garbage collected, so
# we will also store that at the global level as well.
_CACERT_CTX = get_path("certifi", "cacert.pem")
_CACERT_PATH = str(_CACERT_CTX.__enter__())
atexit.register(exit_cacert_ctx)
return _CACERT_PATH
def contents() -> str:
return read_text("certifi", "cacert.pem", encoding="ascii")

0
.venv/lib/python3.12/site-packages/certifi/py.typed Normal file
View File

1
.venv/lib/python3.12/site-packages/charset_normalizer-3.4.7.dist-info/INSTALLER Normal file
View File

@@ -0,0 +1 @@
pip

808
.venv/lib/python3.12/site-packages/charset_normalizer-3.4.7.dist-info/METADATA Normal file
View File

@@ -0,0 +1,808 @@
Metadata-Version: 2.4
Name: charset-normalizer
Version: 3.4.7
Summary: The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
Author-email: "Ahmed R. TAHRI" <tahri.ahmed@proton.me>
Maintainer-email: "Ahmed R. TAHRI" <tahri.ahmed@proton.me>
License: MIT
Project-URL: Changelog, https://github.com/jawah/charset_normalizer/blob/master/CHANGELOG.md
Project-URL: Documentation, https://charset-normalizer.readthedocs.io/
Project-URL: Code, https://github.com/jawah/charset_normalizer
Project-URL: Issue tracker, https://github.com/jawah/charset_normalizer/issues
Keywords: encoding,charset,charset-detector,detector,normalization,unicode,chardet,detect
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Programming Language :: Python :: Free Threading :: 4 - Resilient
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: unicode-backport
Dynamic: license-file
<h1 align="center">Charset Detection, for Everyone 👋</h1>
<p align="center">
<sup>The Real First Universal Charset Detector</sup><br>
<a href="https://pypi.org/project/charset-normalizer">
<img src="https://img.shields.io/pypi/pyversions/charset_normalizer.svg?orange=blue" />
</a>
<a href="https://pepy.tech/project/charset-normalizer/">
<img alt="Download Count Total" src="https://static.pepy.tech/badge/charset-normalizer/month" />
</a>
<a href="https://bestpractices.coreinfrastructure.org/projects/7297">
<img src="https://bestpractices.coreinfrastructure.org/projects/7297/badge">
</a>
</p>
<p align="center">
<sup><i>Featured Packages</i></sup><br>
<a href="https://github.com/jawah/niquests">
<img alt="Static Badge" src="https://img.shields.io/badge/Niquests-Most_Advanced_HTTP_Client-cyan">
</a>
<a href="https://github.com/jawah/wassima">
<img alt="Static Badge" src="https://img.shields.io/badge/Wassima-Certifi_Replacement-cyan">
</a>
</p>
<p align="center">
<sup><i>In other language (unofficial port - by the community)</i></sup><br>
<a href="https://github.com/nickspring/charset-normalizer-rs">
<img alt="Static Badge" src="https://img.shields.io/badge/Rust-red">
</a>
</p>
> A library that helps you read text from an unknown charset encoding.<br /> Motivated by `chardet`,
> I'm trying to resolve the issue by taking a new approach.
> All IANA character set names for which the Python core library provides codecs are supported.
> You can also register your own set of codecs, and yes, it would work as-is.
<p align="center">
>>>>> <a href="https://charsetnormalizerweb.ousret.now.sh" target="_blank">👉 Try Me Online Now, Then Adopt Me 👈 </a> <<<<<
</p>
This project offers you an alternative to **Universal Charset Encoding Detector**, also known as **Chardet**.
| Feature | [Chardet](https://github.com/chardet/chardet) | Charset Normalizer | [cChardet](https://github.com/PyYoshi/cChardet) |
|--------------------------------------------------|:---------------------------------------------:|:-----------------------------------------------------------------------------------------------:|:-----------------------------------------------:|
| `Fast` | ✅ | ✅ | ✅ |
| `Universal`[^1] | ❌ | ✅ | ❌ |
| `Reliable` **without** distinguishable standards | ✅ | ✅ | ✅ |
| `Reliable` **with** distinguishable standards | ✅ | ✅ | ✅ |
| `License` | _Disputed_[^2]<br>_restrictive_ | MIT | MPL-1.1<br>_restrictive_ |
| `Native Python` | ✅ | ✅ | ❌ |
| `Detect spoken language` | ✅ | ✅ | N/A |
| `UnicodeDecodeError Safety` | ✅ | ✅ | ❌ |
| `Whl Size (min)` | 500 kB | 150 kB | ~200 kB |
| `Supported Encoding` | 99 | [99](https://charset-normalizer.readthedocs.io/en/latest/user/support.html#supported-encodings) | 40 |
| `Can register custom encoding` | ❌ | ✅ | ❌ |
<p align="center">
<img src="https://i.imgflip.com/373iay.gif" alt="Reading Normalized Text" width="226"/><img src="https://media.tenor.com/images/c0180f70732a18b4965448d33adba3d0/tenor.gif" alt="Cat Reading Text" width="200"/>
</p>
[^1]: They are clearly using specific code for a specific encoding even if covering most of used one.
[^2]: Chardet 7.0+ was relicensed from LGPL-2.1 to MIT following an AI-assisted rewrite. This relicensing is disputed on two independent grounds: **(a)** the original author [contests](https://github.com/chardet/chardet/issues/327) that the maintainer had the right to relicense, arguing the rewrite is a derivative work of the LGPL-licensed codebase since it was not a clean room implementation; **(b)** the copyright claim itself is [questionable](https://github.com/chardet/chardet/issues/334) given the code was primarily generated by an LLM, and AI-generated output may not be copyrightable under most jurisdictions. Either issue alone could undermine the MIT license. Beyond licensing, the rewrite raises questions about responsible use of AI in open source: key architectural ideas pioneered by charset-normalizer - notably decode-first validity filtering (our foundational approach since v1) and encoding pairwise similarity with the same algorithm and threshold — surfaced in chardet 7 without acknowledgment. The project also imported test files from charset-normalizer to train and benchmark against it, then claimed superior accuracy on those very files. Charset-normalizer has always been MIT-licensed, encoding-agnostic by design, and built on a verifiable human-authored history.
## ⚡ Performance
This package offer better performances (99th, and 95th) against Chardet. Here are some numbers.
| Package | Accuracy | Mean per file (ms) | File per sec (est) |
|---------------------------------------------------|:--------:|:------------------:|:------------------:|
| [chardet 7.1](https://github.com/chardet/chardet) | 89 % | 3 ms | 333 file/sec |
| charset-normalizer | **97 %** | 3 ms | 333 file/sec |
| Package | 99th percentile | 95th percentile | 50th percentile |
|---------------------------------------------------|:---------------:|:---------------:|:---------------:|
| [chardet 7.1](https://github.com/chardet/chardet) | 32 ms | 17 ms | < 1 ms |
| charset-normalizer | 16 ms | 10 ms | 1 ms |
_updated as of March 2026 using CPython 3.12, Charset-Normalizer 3.4.6, and Chardet 7.1.0_
~Chardet's performance on larger file (1MB+) are very poor. Expect huge difference on large payload.~ No longer the case since Chardet 7.0+
> Stats are generated using 400+ files using default parameters. More details on used files, see GHA workflows.
> And yes, these results might change at any time. The dataset can be updated to include more files.
> The actual delays heavily depends on your CPU capabilities. The factors should remain the same.
> Chardet claims on his documentation to have a greater accuracy than us based on the dataset they trained Chardet on(...)
> Well, it's normal, the opposite would have been worrying. Whereas charset-normalizer don't train on anything, our solution
> is based on a completely different algorithm, still heuristic through, it does not need weights across every encoding tables.
## ✨ Installation
Using pip:
```sh
pip install charset-normalizer -U
```
## 🚀 Basic Usage
### CLI
This package comes with a CLI.
```
usage: normalizer [-h] [-v] [-a] [-n] [-m] [-r] [-f] [-t THRESHOLD]
file [file ...]
The Real First Universal Charset Detector. Discover originating encoding used
on text file. Normalize text to unicode.
positional arguments:
files File(s) to be analysed
optional arguments:
-h, --help show this help message and exit
-v, --verbose Display complementary information about file if any.
Stdout will contain logs about the detection process.
-a, --with-alternative
Output complementary possibilities if any. Top-level
JSON WILL be a list.
-n, --normalize Permit to normalize input file. If not set, program
does not write anything.
-m, --minimal Only output the charset detected to STDOUT. Disabling
JSON output.
-r, --replace Replace file when trying to normalize it instead of
creating a new one.
-f, --force Replace file without asking if you are sure, use this
flag with caution.
-t THRESHOLD, --threshold THRESHOLD
Define a custom maximum amount of chaos allowed in
decoded content. 0. <= chaos <= 1.
--version Show version information and exit.
```
```bash
normalizer ./data/sample.1.fr.srt
```
or
```bash
python -m charset_normalizer ./data/sample.1.fr.srt
```
🎉 Since version 1.4.0 the CLI produce easily usable stdout result in JSON format.
```json
{
"path": "/home/default/projects/charset_normalizer/data/sample.1.fr.srt",
"encoding": "cp1252",
"encoding_aliases": [
"1252",
"windows_1252"
],
"alternative_encodings": [
"cp1254",
"cp1256",
"cp1258",
"iso8859_14",
"iso8859_15",
"iso8859_16",
"iso8859_3",
"iso8859_9",
"latin_1",
"mbcs"
],
"language": "French",
"alphabets": [
"Basic Latin",
"Latin-1 Supplement"
],
"has_sig_or_bom": false,
"chaos": 0.149,
"coherence": 97.152,
"unicode_path": null,
"is_preferred": true
}
```
### Python
*Just print out normalized text*
```python
from charset_normalizer import from_path
results = from_path('./my_subtitle.srt')
print(str(results.best()))
```
*Upgrade your code without effort*
```python
from charset_normalizer import detect
```
The above code will behave the same as **chardet**. We ensure that we offer the best (reasonable) BC result possible.
See the docs for advanced usage : [readthedocs.io](https://charset-normalizer.readthedocs.io/en/latest/)
## 😇 Why
When I started using Chardet, I noticed that it was not suited to my expectations, and I wanted to propose a
reliable alternative using a completely different method. Also! I never back down on a good challenge!
I **don't care** about the **originating charset** encoding, because **two different tables** can
produce **two identical rendered string.**
What I want is to get readable text, the best I can.
In a way, **I'm brute forcing text decoding.** How cool is that ? 😎
Don't confuse package **ftfy** with charset-normalizer or chardet. ftfy goal is to repair Unicode string whereas charset-normalizer to convert raw file in unknown encoding to unicode.
## 🍰 How
- Discard all charset encoding table that could not fit the binary content.
- Measure noise, or the mess once opened (by chunks) with a corresponding charset encoding.
- Extract matches with the lowest mess detected.
- Additionally, we measure coherence / probe for a language.
**Wait a minute**, what is noise/mess and coherence according to **YOU ?**
*Noise :* I opened hundred of text files, **written by humans**, with the wrong encoding table. **I observed**, then
**I established** some ground rules about **what is obvious** when **it seems like** a mess (aka. defining noise in rendered text).
I know that my interpretation of what is noise is probably incomplete, feel free to contribute in order to
improve or rewrite it.
*Coherence :* For each language there is on earth, we have computed ranked letter appearance occurrences (the best we can). So I thought
that intel is worth something here. So I use those records against decoded text to check if I can detect intelligent design.
## ⚡ Known limitations
- Language detection is unreliable when text contains two or more languages sharing identical letters. (eg. HTML (english tags) + Turkish content (Sharing Latin characters))
- Every charset detector heavily depends on sufficient content. In common cases, do not bother run detection on very tiny content.
## ⚠️ About Python EOLs
**If you are running:**
- Python >=2.7,<3.5: Unsupported
- Python 3.5: charset-normalizer < 2.1
- Python 3.6: charset-normalizer < 3.1
Upgrade your Python interpreter as soon as possible.
## 👤 Contributing
Contributions, issues and feature requests are very much welcome.<br />
Feel free to check [issues page](https://github.com/ousret/charset_normalizer/issues) if you want to contribute.
## 📝 License
Copyright © [Ahmed TAHRI @Ousret](https://github.com/Ousret).<br />
This project is [MIT](https://github.com/Ousret/charset_normalizer/blob/master/LICENSE) licensed.
Characters frequencies used in this project © 2012 [Denny Vrandečić](http://simia.net/letters/)
## 💼 For Enterprise
Professional support for charset-normalizer is available as part of the [Tidelift
Subscription][1]. Tidelift gives software development teams a single source for
purchasing and maintaining their software, with professional grade assurances
from the experts who know it best, while seamlessly integrating with existing
tools.
[1]: https://tidelift.com/subscription/pkg/pypi-charset-normalizer?utm_source=pypi-charset-normalizer&utm_medium=readme
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/7297/badge)](https://www.bestpractices.dev/projects/7297)
# Changelog
All notable changes to charset-normalizer will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [3.4.7](https://github.com/Ousret/charset_normalizer/compare/3.4.6...3.4.7) (2026-04-02)
### Changed
- Pre-built optimized version using mypy[c] v1.20.
- Relax `setuptools` constraint to `setuptools>=68,<82.1`.
### Fixed
- Correctly remove SIG remnant in utf-7 decoded string. (#718) (#716)
## [3.4.6](https://github.com/Ousret/charset_normalizer/compare/3.4.5...3.4.6) (2026-03-15)
### Changed
- Flattened the logic in `charset_normalizer.md` for higher performance. Removed `eligible(..)` and `feed(...)`
in favor of `feed_info(...)`.
- Raised upper bound for mypy[c] to 1.20, for our optimized version.
- Updated `UNICODE_RANGES_COMBINED` using Unicode blocks v17.
### Fixed
- Edge case where noise difference between two candidates can be almost insignificant. (#672)
- CLI `--normalize` writing to wrong path when passing multiple files in. (#702)
### Misc
- Freethreaded pre-built wheels now shipped in PyPI starting with 3.14t. (#616)
## [3.4.5](https://github.com/Ousret/charset_normalizer/compare/3.4.4...3.4.5) (2026-03-06)
### Changed
- Update `setuptools` constraint to `setuptools>=68,<=82`.
- Raised upper bound of mypyc for the optional pre-built extension to v1.19.1
### Fixed
- Add explicit link to lib math in our optimized build. (#692)
- Logger level not restored correctly for empty byte sequences. (#701)
- TypeError when passing bytearray to from_bytes. (#703)
### Misc
- Applied safe micro-optimizations in both our noise detector and language detector.
- Rewrote the `query_yes_no` function (inside CLI) to avoid using ambiguous licensed code.
- Added `cd.py` submodule into mypyc optional compilation to reduce further the performance impact.
## [3.4.4](https://github.com/Ousret/charset_normalizer/compare/3.4.2...3.4.4) (2025-10-13)
### Changed
- Bound `setuptools` to a specific constraint `setuptools>=68,<=81`.
- Raised upper bound of mypyc for the optional pre-built extension to v1.18.2
### Removed
- `setuptools-scm` as a build dependency.
### Misc
- Enforced hashes in `dev-requirements.txt` and created `ci-requirements.txt` for security purposes.
- Additional pre-built wheels for riscv64, s390x, and armv7l architectures.
- Restore ` multiple.intoto.jsonl` in GitHub releases in addition to individual attestation file per wheel.
## [3.4.3](https://github.com/Ousret/charset_normalizer/compare/3.4.2...3.4.3) (2025-08-09)
### Changed
- mypy(c) is no longer a required dependency at build time if `CHARSET_NORMALIZER_USE_MYPYC` isn't set to `1`. (#595) (#583)
- automatically lower confidence on small bytes samples that are not Unicode in `detect` output legacy function. (#391)
### Added
- Custom build backend to overcome inability to mark mypy as an optional dependency in the build phase.
- Support for Python 3.14
### Fixed
- sdist archive contained useless directories.
- automatically fallback on valid UTF-16 or UTF-32 even if the md says it's noisy. (#633)
### Misc
- SBOM are automatically published to the relevant GitHub release to comply with regulatory changes.
Each published wheel comes with its SBOM. We choose CycloneDX as the format.
- Prebuilt optimized wheel are no longer distributed by default for CPython 3.7 due to a change in cibuildwheel.
## [3.4.2](https://github.com/Ousret/charset_normalizer/compare/3.4.1...3.4.2) (2025-05-02)
### Fixed
- Addressed the DeprecationWarning in our CLI regarding `argparse.FileType` by backporting the target class into the package. (#591)
- Improved the overall reliability of the detector with CJK Ideographs. (#605) (#587)
### Changed
- Optional mypyc compilation upgraded to version 1.15 for Python >= 3.8
## [3.4.1](https://github.com/Ousret/charset_normalizer/compare/3.4.0...3.4.1) (2024-12-24)
### Changed
- Project metadata are now stored using `pyproject.toml` instead of `setup.cfg` using setuptools as the build backend.
- Enforce annotation delayed loading for a simpler and consistent types in the project.
- Optional mypyc compilation upgraded to version 1.14 for Python >= 3.8
### Added
- pre-commit configuration.
- noxfile.
### Removed
- `build-requirements.txt` as per using `pyproject.toml` native build configuration.
- `bin/integration.py` and `bin/serve.py` in favor of downstream integration test (see noxfile).
- `setup.cfg` in favor of `pyproject.toml` metadata configuration.
- Unused `utils.range_scan` function.
### Fixed
- Converting content to Unicode bytes may insert `utf_8` instead of preferred `utf-8`. (#572)
- Deprecation warning "'count' is passed as positional argument" when converting to Unicode bytes on Python 3.13+
## [3.4.0](https://github.com/Ousret/charset_normalizer/compare/3.3.2...3.4.0) (2024-10-08)
### Added
- Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
- Support for Python 3.13 (#512)
### Fixed
- Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
- Improved the general reliability of the detector based on user feedbacks. (#520) (#509) (#498) (#407) (#537)
- Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. (#381)
## [3.3.2](https://github.com/Ousret/charset_normalizer/compare/3.3.1...3.3.2) (2023-10-31)
### Fixed
- Unintentional memory usage regression when using large payload that match several encoding (#376)
- Regression on some detection case showcased in the documentation (#371)
### Added
- Noise (md) probe that identify malformed arabic representation due to the presence of letters in isolated form (credit to my wife)
## [3.3.1](https://github.com/Ousret/charset_normalizer/compare/3.3.0...3.3.1) (2023-10-22)
### Changed
- Optional mypyc compilation upgraded to version 1.6.1 for Python >= 3.8
- Improved the general detection reliability based on reports from the community
## [3.3.0](https://github.com/Ousret/charset_normalizer/compare/3.2.0...3.3.0) (2023-09-30)
### Added
- Allow to execute the CLI (e.g. normalizer) through `python -m charset_normalizer.cli` or `python -m charset_normalizer`
- Support for 9 forgotten encoding that are supported by Python but unlisted in `encoding.aliases` as they have no alias (#323)
### Removed
- (internal) Redundant utils.is_ascii function and unused function is_private_use_only
- (internal) charset_normalizer.assets is moved inside charset_normalizer.constant
### Changed
- (internal) Unicode code blocks in constants are updated using the latest v15.0.0 definition to improve detection
- Optional mypyc compilation upgraded to version 1.5.1 for Python >= 3.8
### Fixed
- Unable to properly sort CharsetMatch when both chaos/noise and coherence were close due to an unreachable condition in \_\_lt\_\_ (#350)
## [3.2.0](https://github.com/Ousret/charset_normalizer/compare/3.1.0...3.2.0) (2023-06-07)
### Changed
- Typehint for function `from_path` no longer enforce `PathLike` as its first argument
- Minor improvement over the global detection reliability
### Added
- Introduce function `is_binary` that relies on main capabilities, and optimized to detect binaries
- Propagate `enable_fallback` argument throughout `from_bytes`, `from_path`, and `from_fp` that allow a deeper control over the detection (default True)
- Explicit support for Python 3.12
### Fixed
- Edge case detection failure where a file would contain 'very-long' camel cased word (Issue #289)
## [3.1.0](https://github.com/Ousret/charset_normalizer/compare/3.0.1...3.1.0) (2023-03-06)
### Added
- Argument `should_rename_legacy` for legacy function `detect` and disregard any new arguments without errors (PR #262)
### Removed
- Support for Python 3.6 (PR #260)
### Changed
- Optional speedup provided by mypy/c 1.0.1
## [3.0.1](https://github.com/Ousret/charset_normalizer/compare/3.0.0...3.0.1) (2022-11-18)
### Fixed
- Multi-bytes cutter/chunk generator did not always cut correctly (PR #233)
### Changed
- Speedup provided by mypy/c 0.990 on Python >= 3.7
## [3.0.0](https://github.com/Ousret/charset_normalizer/compare/2.1.1...3.0.0) (2022-10-20)
### Added
- Extend the capability of explain=True when cp_isolation contains at most two entries (min one), will log in details of the Mess-detector results
- Support for alternative language frequency set in charset_normalizer.assets.FREQUENCIES
- Add parameter `language_threshold` in `from_bytes`, `from_path` and `from_fp` to adjust the minimum expected coherence ratio
- `normalizer --version` now specify if current version provide extra speedup (meaning mypyc compilation whl)
### Changed
- Build with static metadata using 'build' frontend
- Make the language detection stricter
- Optional: Module `md.py` can be compiled using Mypyc to provide an extra speedup up to 4x faster than v2.1
### Fixed
- CLI with opt --normalize fail when using full path for files
- TooManyAccentuatedPlugin induce false positive on the mess detection when too few alpha character have been fed to it
- Sphinx warnings when generating the documentation
### Removed
- Coherence detector no longer return 'Simple English' instead return 'English'
- Coherence detector no longer return 'Classical Chinese' instead return 'Chinese'
- Breaking: Method `first()` and `best()` from CharsetMatch
- UTF-7 will no longer appear as "detected" without a recognized SIG/mark (is unreliable/conflict with ASCII)
- Breaking: Class aliases CharsetDetector, CharsetDoctor, CharsetNormalizerMatch and CharsetNormalizerMatches
- Breaking: Top-level function `normalize`
- Breaking: Properties `chaos_secondary_pass`, `coherence_non_latin` and `w_counter` from CharsetMatch
- Support for the backport `unicodedata2`
## [3.0.0rc1](https://github.com/Ousret/charset_normalizer/compare/3.0.0b2...3.0.0rc1) (2022-10-18)
### Added
- Extend the capability of explain=True when cp_isolation contains at most two entries (min one), will log in details of the Mess-detector results
- Support for alternative language frequency set in charset_normalizer.assets.FREQUENCIES
- Add parameter `language_threshold` in `from_bytes`, `from_path` and `from_fp` to adjust the minimum expected coherence ratio
### Changed
- Build with static metadata using 'build' frontend
- Make the language detection stricter
### Fixed
- CLI with opt --normalize fail when using full path for files
- TooManyAccentuatedPlugin induce false positive on the mess detection when too few alpha character have been fed to it
### Removed
- Coherence detector no longer return 'Simple English' instead return 'English'
- Coherence detector no longer return 'Classical Chinese' instead return 'Chinese'
## [3.0.0b2](https://github.com/Ousret/charset_normalizer/compare/3.0.0b1...3.0.0b2) (2022-08-21)
### Added
- `normalizer --version` now specify if current version provide extra speedup (meaning mypyc compilation whl)
### Removed
- Breaking: Method `first()` and `best()` from CharsetMatch
- UTF-7 will no longer appear as "detected" without a recognized SIG/mark (is unreliable/conflict with ASCII)
### Fixed
- Sphinx warnings when generating the documentation
## [3.0.0b1](https://github.com/Ousret/charset_normalizer/compare/2.1.0...3.0.0b1) (2022-08-15)
### Changed
- Optional: Module `md.py` can be compiled using Mypyc to provide an extra speedup up to 4x faster than v2.1
### Removed
- Breaking: Class aliases CharsetDetector, CharsetDoctor, CharsetNormalizerMatch and CharsetNormalizerMatches
- Breaking: Top-level function `normalize`
- Breaking: Properties `chaos_secondary_pass`, `coherence_non_latin` and `w_counter` from CharsetMatch
- Support for the backport `unicodedata2`
## [2.1.1](https://github.com/Ousret/charset_normalizer/compare/2.1.0...2.1.1) (2022-08-19)
### Deprecated
- Function `normalize` scheduled for removal in 3.0
### Changed
- Removed useless call to decode in fn is_unprintable (#206)
### Fixed
- Third-party library (i18n xgettext) crashing not recognizing utf_8 (PEP 263) with underscore from [@aleksandernovikov](https://github.com/aleksandernovikov) (#204)
## [2.1.0](https://github.com/Ousret/charset_normalizer/compare/2.0.12...2.1.0) (2022-06-19)
### Added
- Output the Unicode table version when running the CLI with `--version` (PR #194)
### Changed
- Re-use decoded buffer for single byte character sets from [@nijel](https://github.com/nijel) (PR #175)
- Fixing some performance bottlenecks from [@deedy5](https://github.com/deedy5) (PR #183)
### Fixed
- Workaround potential bug in cpython with Zero Width No-Break Space located in Arabic Presentation Forms-B, Unicode 1.1 not acknowledged as space (PR #175)
- CLI default threshold aligned with the API threshold from [@oleksandr-kuzmenko](https://github.com/oleksandr-kuzmenko) (PR #181)
### Removed
- Support for Python 3.5 (PR #192)
### Deprecated
- Use of backport unicodedata from `unicodedata2` as Python is quickly catching up, scheduled for removal in 3.0 (PR #194)
## [2.0.12](https://github.com/Ousret/charset_normalizer/compare/2.0.11...2.0.12) (2022-02-12)
### Fixed
- ASCII miss-detection on rare cases (PR #170)
## [2.0.11](https://github.com/Ousret/charset_normalizer/compare/2.0.10...2.0.11) (2022-01-30)
### Added
- Explicit support for Python 3.11 (PR #164)
### Changed
- The logging behavior have been completely reviewed, now using only TRACE and DEBUG levels (PR #163 #165)
## [2.0.10](https://github.com/Ousret/charset_normalizer/compare/2.0.9...2.0.10) (2022-01-04)
### Fixed
- Fallback match entries might lead to UnicodeDecodeError for large bytes sequence (PR #154)
### Changed
- Skipping the language-detection (CD) on ASCII (PR #155)
## [2.0.9](https://github.com/Ousret/charset_normalizer/compare/2.0.8...2.0.9) (2021-12-03)
### Changed
- Moderating the logging impact (since 2.0.8) for specific environments (PR #147)
### Fixed
- Wrong logging level applied when setting kwarg `explain` to True (PR #146)
## [2.0.8](https://github.com/Ousret/charset_normalizer/compare/2.0.7...2.0.8) (2021-11-24)
### Changed
- Improvement over Vietnamese detection (PR #126)
- MD improvement on trailing data and long foreign (non-pure latin) data (PR #124)
- Efficiency improvements in cd/alphabet_languages from [@adbar](https://github.com/adbar) (PR #122)
- call sum() without an intermediary list following PEP 289 recommendations from [@adbar](https://github.com/adbar) (PR #129)
- Code style as refactored by Sourcery-AI (PR #131)
- Minor adjustment on the MD around european words (PR #133)
- Remove and replace SRTs from assets / tests (PR #139)
- Initialize the library logger with a `NullHandler` by default from [@nmaynes](https://github.com/nmaynes) (PR #135)
- Setting kwarg `explain` to True will add provisionally (bounded to function lifespan) a specific stream handler (PR #135)
### Fixed
- Fix large (misleading) sequence giving UnicodeDecodeError (PR #137)
- Avoid using too insignificant chunk (PR #137)
### Added
- Add and expose function `set_logging_handler` to configure a specific StreamHandler from [@nmaynes](https://github.com/nmaynes) (PR #135)
- Add `CHANGELOG.md` entries, format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) (PR #141)
## [2.0.7](https://github.com/Ousret/charset_normalizer/compare/2.0.6...2.0.7) (2021-10-11)
### Added
- Add support for Kazakh (Cyrillic) language detection (PR #109)
### Changed
- Further, improve inferring the language from a given single-byte code page (PR #112)
- Vainly trying to leverage PEP263 when PEP3120 is not supported (PR #116)
- Refactoring for potential performance improvements in loops from [@adbar](https://github.com/adbar) (PR #113)
- Various detection improvement (MD+CD) (PR #117)
### Removed
- Remove redundant logging entry about detected language(s) (PR #115)
### Fixed
- Fix a minor inconsistency between Python 3.5 and other versions regarding language detection (PR #117 #102)
## [2.0.6](https://github.com/Ousret/charset_normalizer/compare/2.0.5...2.0.6) (2021-09-18)
### Fixed
- Unforeseen regression with the loss of the backward-compatibility with some older minor of Python 3.5.x (PR #100)
- Fix CLI crash when using --minimal output in certain cases (PR #103)
### Changed
- Minor improvement to the detection efficiency (less than 1%) (PR #106 #101)
## [2.0.5](https://github.com/Ousret/charset_normalizer/compare/2.0.4...2.0.5) (2021-09-14)
### Changed
- The project now comply with: flake8, mypy, isort and black to ensure a better overall quality (PR #81)
- The BC-support with v1.x was improved, the old staticmethods are restored (PR #82)
- The Unicode detection is slightly improved (PR #93)
- Add syntax sugar \_\_bool\_\_ for results CharsetMatches list-container (PR #91)
### Removed
- The project no longer raise warning on tiny content given for detection, will be simply logged as warning instead (PR #92)
### Fixed
- In some rare case, the chunks extractor could cut in the middle of a multi-byte character and could mislead the mess detection (PR #95)
- Some rare 'space' characters could trip up the UnprintablePlugin/Mess detection (PR #96)
- The MANIFEST.in was not exhaustive (PR #78)
## [2.0.4](https://github.com/Ousret/charset_normalizer/compare/2.0.3...2.0.4) (2021-07-30)
### Fixed
- The CLI no longer raise an unexpected exception when no encoding has been found (PR #70)
- Fix accessing the 'alphabets' property when the payload contains surrogate characters (PR #68)
- The logger could mislead (explain=True) on detected languages and the impact of one MBCS match (PR #72)
- Submatch factoring could be wrong in rare edge cases (PR #72)
- Multiple files given to the CLI were ignored when publishing results to STDOUT. (After the first path) (PR #72)
- Fix line endings from CRLF to LF for certain project files (PR #67)
### Changed
- Adjust the MD to lower the sensitivity, thus improving the global detection reliability (PR #69 #76)
- Allow fallback on specified encoding if any (PR #71)
## [2.0.3](https://github.com/Ousret/charset_normalizer/compare/2.0.2...2.0.3) (2021-07-16)
### Changed
- Part of the detection mechanism has been improved to be less sensitive, resulting in more accurate detection results. Especially ASCII. (PR #63)
- According to the community wishes, the detection will fall back on ASCII or UTF-8 in a last-resort case. (PR #64)
## [2.0.2](https://github.com/Ousret/charset_normalizer/compare/2.0.1...2.0.2) (2021-07-15)
### Fixed
- Empty/Too small JSON payload miss-detection fixed. Report from [@tseaver](https://github.com/tseaver) (PR #59)
### Changed
- Don't inject unicodedata2 into sys.modules from [@akx](https://github.com/akx) (PR #57)
## [2.0.1](https://github.com/Ousret/charset_normalizer/compare/2.0.0...2.0.1) (2021-07-13)
### Fixed
- Make it work where there isn't a filesystem available, dropping assets frequencies.json. Report from [@sethmlarson](https://github.com/sethmlarson). (PR #55)
- Using explain=False permanently disable the verbose output in the current runtime (PR #47)
- One log entry (language target preemptive) was not show in logs when using explain=True (PR #47)
- Fix undesired exception (ValueError) on getitem of instance CharsetMatches (PR #52)
### Changed
- Public function normalize default args values were not aligned with from_bytes (PR #53)
### Added
- You may now use charset aliases in cp_isolation and cp_exclusion arguments (PR #47)
## [2.0.0](https://github.com/Ousret/charset_normalizer/compare/1.4.1...2.0.0) (2021-07-02)
### Changed
- 4x to 5 times faster than the previous 1.4.0 release. At least 2x faster than Chardet.
- Accent has been made on UTF-8 detection, should perform rather instantaneous.
- The backward compatibility with Chardet has been greatly improved. The legacy detect function returns an identical charset name whenever possible.
- The detection mechanism has been slightly improved, now Turkish content is detected correctly (most of the time)
- The program has been rewritten to ease the readability and maintainability. (+Using static typing)+
- utf_7 detection has been reinstated.
### Removed
- This package no longer require anything when used with Python 3.5 (Dropped cached_property)
- Removed support for these languages: Catalan, Esperanto, Kazakh, Baque, Volapük, Azeri, Galician, Nynorsk, Macedonian, and Serbocroatian.
- The exception hook on UnicodeDecodeError has been removed.
### Deprecated
- Methods coherence_non_latin, w_counter, chaos_secondary_pass of the class CharsetMatch are now deprecated and scheduled for removal in v3.0
### Fixed
- The CLI output used the relative path of the file(s). Should be absolute.
## [1.4.1](https://github.com/Ousret/charset_normalizer/compare/1.4.0...1.4.1) (2021-05-28)
### Fixed
- Logger configuration/usage no longer conflict with others (PR #44)
## [1.4.0](https://github.com/Ousret/charset_normalizer/compare/1.3.9...1.4.0) (2021-05-21)
### Removed
- Using standard logging instead of using the package loguru.
- Dropping nose test framework in favor of the maintained pytest.
- Choose to not use dragonmapper package to help with gibberish Chinese/CJK text.
- Require cached_property only for Python 3.5 due to constraint. Dropping for every other interpreter version.
- Stop support for UTF-7 that does not contain a SIG.
- Dropping PrettyTable, replaced with pure JSON output in CLI.
### Fixed
- BOM marker in a CharsetNormalizerMatch instance could be False in rare cases even if obviously present. Due to the sub-match factoring process.
- Not searching properly for the BOM when trying utf32/16 parent codec.
### Changed
- Improving the package final size by compressing frequencies.json.
- Huge improvement over the larges payload.
### Added
- CLI now produces JSON consumable output.
- Return ASCII if given sequences fit. Given reasonable confidence.
## [1.3.9](https://github.com/Ousret/charset_normalizer/compare/1.3.8...1.3.9) (2021-05-13)
### Fixed
- In some very rare cases, you may end up getting encode/decode errors due to a bad bytes payload (PR #40)
## [1.3.8](https://github.com/Ousret/charset_normalizer/compare/1.3.7...1.3.8) (2021-05-12)
### Fixed
- Empty given payload for detection may cause an exception if trying to access the `alphabets` property. (PR #39)
## [1.3.7](https://github.com/Ousret/charset_normalizer/compare/1.3.6...1.3.7) (2021-05-12)
### Fixed
- The legacy detect function should return UTF-8-SIG if sig is present in the payload. (PR #38)
## [1.3.6](https://github.com/Ousret/charset_normalizer/compare/1.3.5...1.3.6) (2021-02-09)
### Changed
- Amend the previous release to allow prettytable 2.0 (PR #35)
## [1.3.5](https://github.com/Ousret/charset_normalizer/compare/1.3.4...1.3.5) (2021-02-08)
### Fixed
- Fix error while using the package with a python pre-release interpreter (PR #33)
### Changed
- Dependencies refactoring, constraints revised.
### Added
- Add python 3.9 and 3.10 to the supported interpreters
MIT License
Copyright (c) 2025 TAHRI Ahmed R.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

36
.venv/lib/python3.12/site-packages/charset_normalizer-3.4.7.dist-info/RECORD Normal file
View File

@@ -0,0 +1,36 @@
../../../bin/normalizer,sha256=n9QFVPfFUaJyWzppQZziJRAHTUy63LNIkD7GHZh70n4,271
81d243bd2c585b0f4821__mypyc.cpython-312-aarch64-linux-gnu.so,sha256=fBcx2RQQPrsMH-GP5rBimq2DFCvcK2q3uvKnoS2jNeg,544080
charset_normalizer-3.4.7.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
charset_normalizer-3.4.7.dist-info/METADATA,sha256=K8lK8L8LaZ1YmKvWLt3zEkpIxiCOC58xNhzFQrfQJxQ,40931
charset_normalizer-3.4.7.dist-info/RECORD,,
charset_normalizer-3.4.7.dist-info/WHEEL,sha256=JdsyDFMymfTgtIbHS9TUHpPfFWZsmU7F9PSqZjXCB2o,193
charset_normalizer-3.4.7.dist-info/entry_points.txt,sha256=ADSTKrkXZ3hhdOVFi6DcUEHQRS0xfxDIE_pEz4wLIXA,65
charset_normalizer-3.4.7.dist-info/licenses/LICENSE,sha256=bQ1Bv-FwrGx9wkjJpj4lTQ-0WmDVCoJX0K-SxuJJuIc,1071
charset_normalizer-3.4.7.dist-info/top_level.txt,sha256=c_vZbitqecT2GfK3zdxSTLCn8C-6pGnHQY5o_5Y32M0,47
charset_normalizer/__init__.py,sha256=OKRxRv2Zhnqk00tqkN0c1BtJjm165fWXLydE52IKuHc,1590
charset_normalizer/__main__.py,sha256=yzYxMR-IhKRHYwcSlavEv8oGdwxsR89mr2X09qXGdps,109
charset_normalizer/__pycache__/__init__.cpython-312.pyc,,
charset_normalizer/__pycache__/__main__.cpython-312.pyc,,
charset_normalizer/__pycache__/api.cpython-312.pyc,,
charset_normalizer/__pycache__/cd.cpython-312.pyc,,
charset_normalizer/__pycache__/constant.cpython-312.pyc,,
charset_normalizer/__pycache__/legacy.cpython-312.pyc,,
charset_normalizer/__pycache__/md.cpython-312.pyc,,
charset_normalizer/__pycache__/models.cpython-312.pyc,,
charset_normalizer/__pycache__/utils.cpython-312.pyc,,
charset_normalizer/__pycache__/version.cpython-312.pyc,,
charset_normalizer/api.py,sha256=387F3n23MlMu-xfSbFULW2DLGsBmVrZVGhnkiGXeKBo,38844
charset_normalizer/cd.cpython-312-aarch64-linux-gnu.so,sha256=EvucwZn9SEgbPs6GqnC_BqqkadD3QEXZ5b5yLR708hY,201304
charset_normalizer/cd.py,sha256=v0iPJweGsRegXywrM1LzUgqW9bJ1KFvIblQHP1jm5FQ,15174
charset_normalizer/cli/__init__.py,sha256=D8I86lFk2-py45JvqxniTirSj_sFyE6sjaY_0-G1shc,136
charset_normalizer/cli/__main__.py,sha256=E9FFSV1E2iOE_B2B1tJHQT9ExJqc60Ks_c-08sNawh8,11940
charset_normalizer/cli/__pycache__/__init__.cpython-312.pyc,,
charset_normalizer/cli/__pycache__/__main__.cpython-312.pyc,,
charset_normalizer/constant.py,sha256=yvLAWDrdSC743Cu4amhwHLIO-FGuRTOTZouCzZKGikc,44431
charset_normalizer/legacy.py,sha256=yBIFMNABNPE5JkdKOWyVo36fZtV9nm8bf37LrDWulz8,2661
charset_normalizer/md.cpython-312-aarch64-linux-gnu.so,sha256=EYzysjLHjG1gl9fj8mFGIRrs0HPeSgwZw5AWcSxih7A,201304
charset_normalizer/md.py,sha256=AYCdfDX79FrgoId3zXqmbCuDcbGr1NRuGqgJN94Rx9Q,30441
charset_normalizer/models.py,sha256=FbaQnI6ECmVmyHRSvVM5fHNeMAQ3KSGdwLjGcQqWDws,12821
charset_normalizer/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
charset_normalizer/utils.py,sha256=9cpi-_0-vC9pGDfuoarhC6VlF_Jxwx5Jsa_8I4w2D8k,12282
charset_normalizer/version.py,sha256=2LxFuGp3BBuIwt95cp64y7v8bCNHcMAi08IfXt_47Co,115

7
.venv/lib/python3.12/site-packages/charset_normalizer-3.4.7.dist-info/WHEEL Normal file
View File

@@ -0,0 +1,7 @@
Wheel-Version: 1.0
Generator: setuptools (82.0.1)
Root-Is-Purelib: false
Tag: cp312-cp312-manylinux_2_17_aarch64
Tag: cp312-cp312-manylinux2014_aarch64
Tag: cp312-cp312-manylinux_2_28_aarch64

2
.venv/lib/python3.12/site-packages/charset_normalizer-3.4.7.dist-info/entry_points.txt Normal file
View File

@@ -0,0 +1,2 @@
[console_scripts]
normalizer = charset_normalizer.cli:cli_detect

21
.venv/lib/python3.12/site-packages/charset_normalizer-3.4.7.dist-info/licenses/LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 TAHRI Ahmed R.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

2
.venv/lib/python3.12/site-packages/charset_normalizer-3.4.7.dist-info/top_level.txt Normal file
View File

@@ -0,0 +1,2 @@
81d243bd2c585b0f4821__mypyc
charset_normalizer

48
.venv/lib/python3.12/site-packages/charset_normalizer/__init__.py Normal file
View File

@@ -0,0 +1,48 @@
"""
Charset-Normalizer
~~~~~~~~~~~~~~
The Real First Universal Charset Detector.
A library that helps you read text from an unknown charset encoding.
Motivated by chardet, This package is trying to resolve the issue by taking a new approach.
All IANA character set names for which the Python core library provides codecs are supported.
Basic usage:
>>> from charset_normalizer import from_bytes
>>> results = from_bytes('Bсеки човек има право на образование. Oбразованието!'.encode('utf_8'))
>>> best_guess = results.best()
>>> str(best_guess)
'Bсеки човек има право на образование. Oбразованието!'
Others methods and usages are available - see the full documentation
at <https://github.com/Ousret/charset_normalizer>.
:copyright: (c) 2021 by Ahmed TAHRI
:license: MIT, see LICENSE for more details.
"""
from __future__ import annotations
import logging
from .api import from_bytes, from_fp, from_path, is_binary
from .legacy import detect
from .models import CharsetMatch, CharsetMatches
from .utils import set_logging_handler
from .version import VERSION, __version__
__all__ = (
"from_fp",
"from_path",
"from_bytes",
"is_binary",
"detect",
"CharsetMatch",
"CharsetMatches",
"__version__",
"VERSION",
"set_logging_handler",
)
# Attach a NullHandler to the top level logger by default
# https://docs.python.org/3.3/howto/logging.html#configuring-logging-for-a-library
logging.getLogger("charset_normalizer").addHandler(logging.NullHandler())

6
.venv/lib/python3.12/site-packages/charset_normalizer/__main__.py Normal file
View File

@@ -0,0 +1,6 @@
from __future__ import annotations
from .cli import cli_detect
if __name__ == "__main__":
cli_detect()

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/__init__.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/__main__.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/api.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/cd.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/constant.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/legacy.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/md.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/models.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/utils.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/__pycache__/version.cpython-312.pyc Normal file
View File

Binary file not shown.

988
.venv/lib/python3.12/site-packages/charset_normalizer/api.py Normal file
View File

@@ -0,0 +1,988 @@
from __future__ import annotations
import logging
from os import PathLike
from typing import BinaryIO
from .cd import (
coherence_ratio,
encoding_languages,
mb_encoding_languages,
merge_coherence_ratios,
)
from .constant import (
IANA_SUPPORTED,
IANA_SUPPORTED_SIMILAR,
TOO_BIG_SEQUENCE,
TOO_SMALL_SEQUENCE,
TRACE,
)
from .md import mess_ratio
from .models import CharsetMatch, CharsetMatches
from .utils import (
any_specified_encoding,
cut_sequence_chunks,
iana_name,
identify_sig_or_bom,
is_multi_byte_encoding,
should_strip_sig_or_bom,
)
logger = logging.getLogger("charset_normalizer")
explain_handler = logging.StreamHandler()
explain_handler.setFormatter(
logging.Formatter("%(asctime)s | %(levelname)s | %(message)s")
)
# Pre-compute a reordered encoding list: multibyte first, then single-byte.
# This allows the mb_definitive_match optimization to fire earlier, skipping
# all single-byte encodings for genuine CJK content. Multibyte codecs
# hard-fail (UnicodeDecodeError) on single-byte data almost instantly, so
# testing them first costs negligible time for non-CJK files.
_mb_supported: list[str] = []
_sb_supported: list[str] = []
for _supported_enc in IANA_SUPPORTED:
try:
if is_multi_byte_encoding(_supported_enc):
_mb_supported.append(_supported_enc)
else:
_sb_supported.append(_supported_enc)
except ImportError:
_sb_supported.append(_supported_enc)
IANA_SUPPORTED_MB_FIRST: list[str] = _mb_supported + _sb_supported
def from_bytes(
sequences: bytes | bytearray,
steps: int = 5,
chunk_size: int = 512,
threshold: float = 0.2,
cp_isolation: list[str] | None = None,
cp_exclusion: list[str] | None = None,
preemptive_behaviour: bool = True,
explain: bool = False,
language_threshold: float = 0.1,
enable_fallback: bool = True,
) -> CharsetMatches:
"""
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
If there is no results, it is a strong indicator that the source is binary/not text.
By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence.
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
but never take it for granted. Can improve the performance.
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
purpose.
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
Custom logging format and handler can be set manually.
"""
if not isinstance(sequences, (bytearray, bytes)):
raise TypeError(
"Expected object of type bytes or bytearray, got: {}".format(
type(sequences)
)
)
if explain:
previous_logger_level: int = logger.level
logger.addHandler(explain_handler)
logger.setLevel(TRACE)
length: int = len(sequences)
if length == 0:
logger.debug("Encoding detection on empty bytes, assuming utf_8 intention.")
if explain: # Defensive: ensure exit path clean handler
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([CharsetMatch(sequences, "utf_8", 0.0, False, [], "")])
if cp_isolation is not None:
logger.log(
TRACE,
"cp_isolation is set. use this flag for debugging purpose. "
"limited list of encoding allowed : %s.",
", ".join(cp_isolation),
)
cp_isolation = [iana_name(cp, False) for cp in cp_isolation]
else:
cp_isolation = []
if cp_exclusion is not None:
logger.log(
TRACE,
"cp_exclusion is set. use this flag for debugging purpose. "
"limited list of encoding excluded : %s.",
", ".join(cp_exclusion),
)
cp_exclusion = [iana_name(cp, False) for cp in cp_exclusion]
else:
cp_exclusion = []
if length <= (chunk_size * steps):
logger.log(
TRACE,
"override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.",
steps,
chunk_size,
length,
)
steps = 1
chunk_size = length
if steps > 1 and length / steps < chunk_size:
chunk_size = int(length / steps)
is_too_small_sequence: bool = len(sequences) < TOO_SMALL_SEQUENCE
is_too_large_sequence: bool = len(sequences) >= TOO_BIG_SEQUENCE
if is_too_small_sequence:
logger.log(
TRACE,
"Trying to detect encoding from a tiny portion of ({}) byte(s).".format(
length
),
)
elif is_too_large_sequence:
logger.log(
TRACE,
"Using lazy str decoding because the payload is quite large, ({}) byte(s).".format(
length
),
)
prioritized_encodings: list[str] = []
specified_encoding: str | None = (
any_specified_encoding(sequences) if preemptive_behaviour else None
)
if specified_encoding is not None:
prioritized_encodings.append(specified_encoding)
logger.log(
TRACE,
"Detected declarative mark in sequence. Priority +1 given for %s.",
specified_encoding,
)
tested: set[str] = set()
tested_but_hard_failure: list[str] = []
tested_but_soft_failure: list[str] = []
soft_failure_skip: set[str] = set()
success_fast_tracked: set[str] = set()
# Cache for decoded payload deduplication: hash(decoded_payload) -> (mean_mess_ratio, cd_ratios_merged, passed)
# When multiple encodings decode to the exact same string, we can skip the expensive
# mess_ratio and coherence_ratio analysis and reuse the results from the first encoding.
payload_result_cache: dict[int, tuple[float, list[tuple[str, float]], bool]] = {}
# When a definitive result (chaos=0.0 and good coherence) is found after testing
# the prioritized encodings (ascii, utf_8), we can significantly reduce the remaining
# work. Encodings that target completely different language families (e.g., Cyrillic
# when the definitive match is Latin) are skipped entirely.
# Additionally, for same-family encodings that pass chaos probing, we reuse the
# definitive match's coherence ratios instead of recomputing them — a major savings
# since coherence_ratio accounts for ~30% of total time on slow Latin files.
definitive_match_found: bool = False
definitive_target_languages: set[str] = set()
# After the definitive match fires, we cap the number of additional same-family
# single-byte encodings that pass chaos probing. Once we've accumulated enough
# good candidates (N), further same-family SB encodings are unlikely to produce
# a better best() result and just waste mess_ratio + coherence_ratio time.
# The first encoding to trigger the definitive match is NOT counted (it's already in).
post_definitive_sb_success_count: int = 0
POST_DEFINITIVE_SB_CAP: int = 7
# When a non-UTF multibyte encoding passes chaos probing with significant multibyte
# content (decoded length < 98% of raw length), skip all remaining single-byte encodings.
# Rationale: multi-byte decoders (CJK) have strict byte-sequence validation — if they
# decode without error AND pass chaos probing with substantial multibyte content, the
# data is genuinely multibyte encoded. Single-byte encodings will always decode (every
# byte maps to something) but waste time on mess_ratio before failing.
# The 98% threshold prevents false triggers on files that happen to have a few valid
# multibyte pairs (e.g., cp424/_ude_1.txt where big5 decodes with 99% ratio).
mb_definitive_match_found: bool = False
fallback_ascii: CharsetMatch | None = None
fallback_u8: CharsetMatch | None = None
fallback_specified: CharsetMatch | None = None
results: CharsetMatches = CharsetMatches()
early_stop_results: CharsetMatches = CharsetMatches()
sig_encoding, sig_payload = identify_sig_or_bom(sequences)
if sig_encoding is not None:
prioritized_encodings.append(sig_encoding)
logger.log(
TRACE,
"Detected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.",
len(sig_payload),
sig_encoding,
)
prioritized_encodings.append("ascii")
if "utf_8" not in prioritized_encodings:
prioritized_encodings.append("utf_8")
for encoding_iana in prioritized_encodings + IANA_SUPPORTED_MB_FIRST:
if cp_isolation and encoding_iana not in cp_isolation:
continue
if cp_exclusion and encoding_iana in cp_exclusion:
continue
if encoding_iana in tested:
continue
tested.add(encoding_iana)
decoded_payload: str | None = None
bom_or_sig_available: bool = sig_encoding == encoding_iana
strip_sig_or_bom: bool = bom_or_sig_available and should_strip_sig_or_bom(
encoding_iana
)
if encoding_iana in {"utf_16", "utf_32"} and not bom_or_sig_available:
logger.log(
TRACE,
"Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.",
encoding_iana,
)
continue
if encoding_iana in {"utf_7"} and not bom_or_sig_available:
logger.log(
TRACE,
"Encoding %s won't be tested as-is because detection is unreliable without BOM/SIG.",
encoding_iana,
)
continue
# Skip encodings similar to ones that already soft-failed (high mess ratio).
# Checked BEFORE the expensive decode attempt.
if encoding_iana in soft_failure_skip:
logger.log(
TRACE,
"%s is deemed too similar to a code page that was already considered unsuited. Continuing!",
encoding_iana,
)
continue
# Skip encodings that were already fast-tracked from a similar successful encoding.
if encoding_iana in success_fast_tracked:
logger.log(
TRACE,
"Skipping %s: already fast-tracked from a similar successful encoding.",
encoding_iana,
)
continue
try:
is_multi_byte_decoder: bool = is_multi_byte_encoding(encoding_iana)
except (ModuleNotFoundError, ImportError): # Defensive:
logger.log(
TRACE,
"Encoding %s does not provide an IncrementalDecoder",
encoding_iana,
)
continue
# When we've already found a definitive match (chaos=0.0 with good coherence)
# after testing the prioritized encodings, skip encodings that target
# completely different language families. This avoids running expensive
# mess_ratio + coherence_ratio on clearly unrelated candidates (e.g., Cyrillic
# when the definitive match is Latin-based).
if definitive_match_found:
if not is_multi_byte_decoder:
enc_languages = set(encoding_languages(encoding_iana))
else:
enc_languages = set(mb_encoding_languages(encoding_iana))
if not enc_languages.intersection(definitive_target_languages):
logger.log(
TRACE,
"Skipping %s: definitive match already found, this encoding targets different languages (%s vs %s).",
encoding_iana,
enc_languages,
definitive_target_languages,
)
continue
# After the definitive match, cap the number of additional same-family
# single-byte encodings that pass chaos probing. This avoids testing the
# tail of rare, low-value same-family encodings (mac_iceland, cp860, etc.)
# that almost never change best() but each cost ~1-2ms of mess_ratio + coherence.
if (
definitive_match_found
and not is_multi_byte_decoder
and post_definitive_sb_success_count >= POST_DEFINITIVE_SB_CAP
):
logger.log(
TRACE,
"Skipping %s: already accumulated %d same-family results after definitive match (cap=%d).",
encoding_iana,
post_definitive_sb_success_count,
POST_DEFINITIVE_SB_CAP,
)
continue
# When a multibyte encoding with significant multibyte content has already
# passed chaos probing, skip all single-byte encodings. They will either fail
# chaos probing (wasting mess_ratio time) or produce inferior results.
if mb_definitive_match_found and not is_multi_byte_decoder:
logger.log(
TRACE,
"Skipping single-byte %s: multi-byte definitive match already found.",
encoding_iana,
)
continue
try:
if is_too_large_sequence and is_multi_byte_decoder is False:
str(
(
sequences[: int(50e4)]
if strip_sig_or_bom is False
else sequences[len(sig_payload) : int(50e4)]
),
encoding=encoding_iana,
)
else:
# UTF-7 BOM is encoded in modified Base64 whose byte boundary
# can overlap with the next character. Stripping raw SIG bytes
# before decoding may leave stray bytes that decode as garbage.
# Decode the full sequence and remove the leading BOM char instead.
# see https://github.com/jawah/charset_normalizer/issues/718
# and https://github.com/jawah/charset_normalizer/issues/716
if encoding_iana == "utf_7" and bom_or_sig_available:
decoded_payload = str(
sequences,
encoding=encoding_iana,
)
if decoded_payload and decoded_payload[0] == "\ufeff":
decoded_payload = decoded_payload[1:]
else:
decoded_payload = str(
(
sequences
if strip_sig_or_bom is False
else sequences[len(sig_payload) :]
),
encoding=encoding_iana,
)
except (UnicodeDecodeError, LookupError) as e:
if not isinstance(e, LookupError):
logger.log(
TRACE,
"Code page %s does not fit given bytes sequence at ALL. %s",
encoding_iana,
str(e),
)
tested_but_hard_failure.append(encoding_iana)
continue
r_ = range(
0 if not bom_or_sig_available else len(sig_payload),
length,
int(length / steps),
)
multi_byte_bonus: bool = (
is_multi_byte_decoder
and decoded_payload is not None
and len(decoded_payload) < length
)
if multi_byte_bonus:
logger.log(
TRACE,
"Code page %s is a multi byte encoding table and it appear that at least one character "
"was encoded using n-bytes.",
encoding_iana,
)
# Payload-hash deduplication: if another encoding already decoded to the
# exact same string, reuse its mess_ratio and coherence results entirely.
# This is strictly more general than the old IANA_SUPPORTED_SIMILAR approach
# because it catches ALL identical decoding, not just pre-mapped ones.
if decoded_payload is not None and not is_multi_byte_decoder:
payload_hash: int = hash(decoded_payload)
cached = payload_result_cache.get(payload_hash)
if cached is not None:
cached_mess, cached_cd, cached_passed = cached
if cached_passed:
# The previous encoding with identical output passed chaos probing.
fast_match = CharsetMatch(
sequences,
encoding_iana,
cached_mess,
bom_or_sig_available,
cached_cd,
(
decoded_payload
if (
is_too_large_sequence is False
or encoding_iana
in [specified_encoding, "ascii", "utf_8"]
)
else None
),
preemptive_declaration=specified_encoding,
)
results.append(fast_match)
success_fast_tracked.add(encoding_iana)
logger.log(
TRACE,
"%s fast-tracked (identical decoded payload to a prior encoding, chaos=%f %%).",
encoding_iana,
round(cached_mess * 100, ndigits=3),
)
if (
encoding_iana in [specified_encoding, "ascii", "utf_8"]
and cached_mess < 0.1
):
if cached_mess == 0.0:
logger.debug(
"Encoding detection: %s is most likely the one.",
fast_match.encoding,
)
if explain:
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([fast_match])
early_stop_results.append(fast_match)
if (
len(early_stop_results)
and (specified_encoding is None or specified_encoding in tested)
and "ascii" in tested
and "utf_8" in tested
):
probable_result: CharsetMatch = early_stop_results.best() # type: ignore[assignment]
logger.debug(
"Encoding detection: %s is most likely the one.",
probable_result.encoding,
)
if explain:
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([probable_result])
continue
else:
# The previous encoding with identical output failed chaos probing.
tested_but_soft_failure.append(encoding_iana)
logger.log(
TRACE,
"%s fast-skipped (identical decoded payload to a prior encoding that failed chaos probing).",
encoding_iana,
)
# Prepare fallbacks for special encodings even when skipped.
if enable_fallback and encoding_iana in [
"ascii",
"utf_8",
specified_encoding,
"utf_16",
"utf_32",
]:
fallback_entry = CharsetMatch(
sequences,
encoding_iana,
threshold,
bom_or_sig_available,
[],
decoded_payload,
preemptive_declaration=specified_encoding,
)
if encoding_iana == specified_encoding:
fallback_specified = fallback_entry
elif encoding_iana == "ascii":
fallback_ascii = fallback_entry
else:
fallback_u8 = fallback_entry
continue
max_chunk_gave_up: int = int(len(r_) / 4)
max_chunk_gave_up = max(max_chunk_gave_up, 2)
early_stop_count: int = 0
lazy_str_hard_failure = False
md_chunks: list[str] = []
md_ratios = []
try:
for chunk in cut_sequence_chunks(
sequences,
encoding_iana,
r_,
chunk_size,
bom_or_sig_available,
strip_sig_or_bom,
sig_payload,
is_multi_byte_decoder,
decoded_payload,
):
md_chunks.append(chunk)
md_ratios.append(
mess_ratio(
chunk,
threshold,
explain is True and 1 <= len(cp_isolation) <= 2,
)
)
if md_ratios[-1] >= threshold:
early_stop_count += 1
if (early_stop_count >= max_chunk_gave_up) or (
bom_or_sig_available and strip_sig_or_bom is False
):
break
except (
UnicodeDecodeError
) as e: # Lazy str loading may have missed something there
logger.log(
TRACE,
"LazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %s",
encoding_iana,
str(e),
)
early_stop_count = max_chunk_gave_up
lazy_str_hard_failure = True
# We might want to check the sequence again with the whole content
# Only if initial MD tests passes
if (
not lazy_str_hard_failure
and is_too_large_sequence
and not is_multi_byte_decoder
):
try:
sequences[int(50e3) :].decode(encoding_iana, errors="strict")
except UnicodeDecodeError as e:
logger.log(
TRACE,
"LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %s",
encoding_iana,
str(e),
)
tested_but_hard_failure.append(encoding_iana)
continue
mean_mess_ratio: float = sum(md_ratios) / len(md_ratios) if md_ratios else 0.0
if mean_mess_ratio >= threshold or early_stop_count >= max_chunk_gave_up:
tested_but_soft_failure.append(encoding_iana)
if encoding_iana in IANA_SUPPORTED_SIMILAR:
soft_failure_skip.update(IANA_SUPPORTED_SIMILAR[encoding_iana])
# Cache this soft-failure so identical decoding from other encodings
# can be skipped immediately.
if decoded_payload is not None and not is_multi_byte_decoder:
payload_result_cache.setdefault(
hash(decoded_payload), (mean_mess_ratio, [], False)
)
logger.log(
TRACE,
"%s was excluded because of initial chaos probing. Gave up %i time(s). "
"Computed mean chaos is %f %%.",
encoding_iana,
early_stop_count,
round(mean_mess_ratio * 100, ndigits=3),
)
# Preparing those fallbacks in case we got nothing.
if (
enable_fallback
and encoding_iana
in ["ascii", "utf_8", specified_encoding, "utf_16", "utf_32"]
and not lazy_str_hard_failure
):
fallback_entry = CharsetMatch(
sequences,
encoding_iana,
threshold,
bom_or_sig_available,
[],
decoded_payload,
preemptive_declaration=specified_encoding,
)
if encoding_iana == specified_encoding:
fallback_specified = fallback_entry
elif encoding_iana == "ascii":
fallback_ascii = fallback_entry
else:
fallback_u8 = fallback_entry
continue
logger.log(
TRACE,
"%s passed initial chaos probing. Mean measured chaos is %f %%",
encoding_iana,
round(mean_mess_ratio * 100, ndigits=3),
)
if not is_multi_byte_decoder:
target_languages: list[str] = encoding_languages(encoding_iana)
else:
target_languages = mb_encoding_languages(encoding_iana)
if target_languages:
logger.log(
TRACE,
"{} should target any language(s) of {}".format(
encoding_iana, str(target_languages)
),
)
cd_ratios = []
# Run coherence detection on all chunks. We previously tried limiting to
# 1-2 chunks for post-definitive encodings to save time, but this caused
# coverage regressions by producing unrepresentative coherence scores.
# The SB cap and language-family skip optimizations provide sufficient
# speedup without sacrificing coherence accuracy.
if encoding_iana != "ascii":
# We shall skip the CD when its about ASCII
# Most of the time its not relevant to run "language-detection" on it.
for chunk in md_chunks:
chunk_languages = coherence_ratio(
chunk,
language_threshold,
",".join(target_languages) if target_languages else None,
)
cd_ratios.append(chunk_languages)
cd_ratios_merged = merge_coherence_ratios(cd_ratios)
else:
cd_ratios_merged = merge_coherence_ratios(cd_ratios)
if cd_ratios_merged:
logger.log(
TRACE,
"We detected language {} using {}".format(
cd_ratios_merged, encoding_iana
),
)
current_match = CharsetMatch(
sequences,
encoding_iana,
mean_mess_ratio,
bom_or_sig_available,
cd_ratios_merged,
(
decoded_payload
if (
is_too_large_sequence is False
or encoding_iana in [specified_encoding, "ascii", "utf_8"]
)
else None
),
preemptive_declaration=specified_encoding,
)
results.append(current_match)
# Cache the successful result for payload-hash deduplication.
if decoded_payload is not None and not is_multi_byte_decoder:
payload_result_cache.setdefault(
hash(decoded_payload),
(mean_mess_ratio, cd_ratios_merged, True),
)
# Count post-definitive same-family SB successes for the early termination cap.
# Only count low-mess encodings (< 2%) toward the cap. High-mess encodings are
# marginal results that shouldn't prevent better-quality candidates from being
# tested. For example, iso8859_4 (mess=0%) should not be skipped just because
# 7 high-mess Latin encodings (cp1252 at 8%, etc.) were tried first.
if (
definitive_match_found
and not is_multi_byte_decoder
and mean_mess_ratio < 0.02
):
post_definitive_sb_success_count += 1
if (
encoding_iana in [specified_encoding, "ascii", "utf_8"]
and mean_mess_ratio < 0.1
):
# If md says nothing to worry about, then... stop immediately!
if mean_mess_ratio == 0.0:
logger.debug(
"Encoding detection: %s is most likely the one.",
current_match.encoding,
)
if explain: # Defensive: ensure exit path clean handler
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([current_match])
early_stop_results.append(current_match)
if (
len(early_stop_results)
and (specified_encoding is None or specified_encoding in tested)
and "ascii" in tested
and "utf_8" in tested
):
probable_result = early_stop_results.best() # type: ignore[assignment]
logger.debug(
"Encoding detection: %s is most likely the one.",
probable_result.encoding, # type: ignore[union-attr]
)
if explain: # Defensive: ensure exit path clean handler
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([probable_result])
# Once we find a result with good coherence (>= 0.5) after testing the
# prioritized encodings (ascii, utf_8), activate "definitive mode": skip
# encodings that target completely different language families. This avoids
# running expensive mess_ratio + coherence_ratio on clearly unrelated
# candidates (e.g., Cyrillic encodings when the match is Latin-based).
# We require coherence >= 0.5 to avoid false positives (e.g., cp1251 decoding
# Hebrew text with 0.0 chaos but wrong language detection at coherence 0.33).
if not definitive_match_found and not is_multi_byte_decoder:
best_coherence = (
max((v for _, v in cd_ratios_merged), default=0.0)
if cd_ratios_merged
else 0.0
)
if best_coherence >= 0.5 and "ascii" in tested and "utf_8" in tested:
definitive_match_found = True
definitive_target_languages.update(target_languages)
logger.log(
TRACE,
"Definitive match found: %s (chaos=%.3f, coherence=%.2f). Encodings targeting different language families will be skipped.",
encoding_iana,
mean_mess_ratio,
best_coherence,
)
# When a non-UTF multibyte encoding passes chaos probing with significant
# multibyte content (decoded < 98% of raw), activate mb_definitive_match.
# This skips all remaining single-byte encodings which would either soft-fail
# (running expensive mess_ratio for nothing) or produce inferior results.
if (
not mb_definitive_match_found
and is_multi_byte_decoder
and multi_byte_bonus
and decoded_payload is not None
and len(decoded_payload) < length * 0.98
and encoding_iana
not in {
"utf_8",
"utf_8_sig",
"utf_16",
"utf_16_be",
"utf_16_le",
"utf_32",
"utf_32_be",
"utf_32_le",
"utf_7",
}
and "ascii" in tested
and "utf_8" in tested
):
mb_definitive_match_found = True
logger.log(
TRACE,
"Multi-byte definitive match: %s (chaos=%.3f, decoded=%d/%d=%.1f%%). Single-byte encodings will be skipped.",
encoding_iana,
mean_mess_ratio,
len(decoded_payload),
length,
len(decoded_payload) / length * 100,
)
if encoding_iana == sig_encoding:
logger.debug(
"Encoding detection: %s is most likely the one as we detected a BOM or SIG within "
"the beginning of the sequence.",
encoding_iana,
)
if explain: # Defensive: ensure exit path clean handler
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([results[encoding_iana]])
if len(results) == 0:
if fallback_u8 or fallback_ascii or fallback_specified:
logger.log(
TRACE,
"Nothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.",
)
if fallback_specified:
logger.debug(
"Encoding detection: %s will be used as a fallback match",
fallback_specified.encoding,
)
results.append(fallback_specified)
elif (
(fallback_u8 and fallback_ascii is None)
or (
fallback_u8
and fallback_ascii
and fallback_u8.fingerprint != fallback_ascii.fingerprint
)
or (fallback_u8 is not None)
):
logger.debug("Encoding detection: utf_8 will be used as a fallback match")
results.append(fallback_u8)
elif fallback_ascii:
logger.debug("Encoding detection: ascii will be used as a fallback match")
results.append(fallback_ascii)
if results:
logger.debug(
"Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.",
results.best().encoding, # type: ignore
len(results) - 1,
)
else:
logger.debug("Encoding detection: Unable to determine any suitable charset.")
if explain:
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return results
def from_fp(
fp: BinaryIO,
steps: int = 5,
chunk_size: int = 512,
threshold: float = 0.20,
cp_isolation: list[str] | None = None,
cp_exclusion: list[str] | None = None,
preemptive_behaviour: bool = True,
explain: bool = False,
language_threshold: float = 0.1,
enable_fallback: bool = True,
) -> CharsetMatches:
"""
Same thing than the function from_bytes but using a file pointer that is already ready.
Will not close the file pointer.
"""
return from_bytes(
fp.read(),
steps,
chunk_size,
threshold,
cp_isolation,
cp_exclusion,
preemptive_behaviour,
explain,
language_threshold,
enable_fallback,
)
def from_path(
path: str | bytes | PathLike, # type: ignore[type-arg]
steps: int = 5,
chunk_size: int = 512,
threshold: float = 0.20,
cp_isolation: list[str] | None = None,
cp_exclusion: list[str] | None = None,
preemptive_behaviour: bool = True,
explain: bool = False,
language_threshold: float = 0.1,
enable_fallback: bool = True,
) -> CharsetMatches:
"""
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
Can raise IOError.
"""
with open(path, "rb") as fp:
return from_fp(
fp,
steps,
chunk_size,
threshold,
cp_isolation,
cp_exclusion,
preemptive_behaviour,
explain,
language_threshold,
enable_fallback,
)
def is_binary(
fp_or_path_or_payload: PathLike | str | BinaryIO | bytes, # type: ignore[type-arg]
steps: int = 5,
chunk_size: int = 512,
threshold: float = 0.20,
cp_isolation: list[str] | None = None,
cp_exclusion: list[str] | None = None,
preemptive_behaviour: bool = True,
explain: bool = False,
language_threshold: float = 0.1,
enable_fallback: bool = False,
) -> bool:
"""
Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string.
Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match
are disabled to be stricter around ASCII-compatible but unlikely to be a string.
"""
if isinstance(fp_or_path_or_payload, (str, PathLike)):
guesses = from_path(
fp_or_path_or_payload,
steps=steps,
chunk_size=chunk_size,
threshold=threshold,
cp_isolation=cp_isolation,
cp_exclusion=cp_exclusion,
preemptive_behaviour=preemptive_behaviour,
explain=explain,
language_threshold=language_threshold,
enable_fallback=enable_fallback,
)
elif isinstance(
fp_or_path_or_payload,
(
bytes,
bytearray,
),
):
guesses = from_bytes(
fp_or_path_or_payload,
steps=steps,
chunk_size=chunk_size,
threshold=threshold,
cp_isolation=cp_isolation,
cp_exclusion=cp_exclusion,
preemptive_behaviour=preemptive_behaviour,
explain=explain,
language_threshold=language_threshold,
enable_fallback=enable_fallback,
)
else:
guesses = from_fp(
fp_or_path_or_payload,
steps=steps,
chunk_size=chunk_size,
threshold=threshold,
cp_isolation=cp_isolation,
cp_exclusion=cp_exclusion,
preemptive_behaviour=preemptive_behaviour,
explain=explain,
language_threshold=language_threshold,
enable_fallback=enable_fallback,
)
return not guesses

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/cd.cpython-312-aarch64-linux-gnu.so Executable file
View File

Binary file not shown.

454
.venv/lib/python3.12/site-packages/charset_normalizer/cd.py Normal file
View File

@@ -0,0 +1,454 @@
from __future__ import annotations
import importlib
from codecs import IncrementalDecoder
from collections import Counter
from functools import lru_cache
from typing import Counter as TypeCounter
from .constant import (
FREQUENCIES,
KO_NAMES,
LANGUAGE_SUPPORTED_COUNT,
TOO_SMALL_SEQUENCE,
ZH_NAMES,
_FREQUENCIES_SET,
_FREQUENCIES_RANK,
)
from .md import is_suspiciously_successive_range
from .models import CoherenceMatches
from .utils import (
is_accentuated,
is_latin,
is_multi_byte_encoding,
is_unicode_range_secondary,
unicode_range,
)
def encoding_unicode_range(iana_name: str) -> list[str]:
"""
Return associated unicode ranges in a single byte code page.
"""
if is_multi_byte_encoding(iana_name):
raise OSError( # Defensive:
"Function not supported on multi-byte code page"
)
decoder = importlib.import_module(f"encodings.{iana_name}").IncrementalDecoder
p: IncrementalDecoder = decoder(errors="ignore")
seen_ranges: dict[str, int] = {}
character_count: int = 0
for i in range(0x40, 0xFF):
chunk: str = p.decode(bytes([i]))
if chunk:
character_range: str | None = unicode_range(chunk)
if character_range is None:
continue
if is_unicode_range_secondary(character_range) is False:
if character_range not in seen_ranges:
seen_ranges[character_range] = 0
seen_ranges[character_range] += 1
character_count += 1
return sorted(
[
character_range
for character_range in seen_ranges
if seen_ranges[character_range] / character_count >= 0.15
]
)
def unicode_range_languages(primary_range: str) -> list[str]:
"""
Return inferred languages used with a unicode range.
"""
languages: list[str] = []
for language, characters in FREQUENCIES.items():
for character in characters:
if unicode_range(character) == primary_range:
languages.append(language)
break
return languages
@lru_cache()
def encoding_languages(iana_name: str) -> list[str]:
"""
Single-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
"""
unicode_ranges: list[str] = encoding_unicode_range(iana_name)
primary_range: str | None = None
for specified_range in unicode_ranges:
if "Latin" not in specified_range:
primary_range = specified_range
break
if primary_range is None:
return ["Latin Based"]
return unicode_range_languages(primary_range)
@lru_cache()
def mb_encoding_languages(iana_name: str) -> list[str]:
"""
Multi-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
"""
if (
iana_name.startswith("shift_")
or iana_name.startswith("iso2022_jp")
or iana_name.startswith("euc_j")
or iana_name == "cp932"
):
return ["Japanese"]
if iana_name.startswith("gb") or iana_name in ZH_NAMES:
return ["Chinese"]
if iana_name.startswith("iso2022_kr") or iana_name in KO_NAMES:
return ["Korean"]
return []
@lru_cache(maxsize=LANGUAGE_SUPPORTED_COUNT)
def get_target_features(language: str) -> tuple[bool, bool]:
"""
Determine main aspects from a supported language if it contains accents and if is pure Latin.
"""
target_have_accents: bool = False
target_pure_latin: bool = True
for character in FREQUENCIES[language]:
if not target_have_accents and is_accentuated(character):
target_have_accents = True
if target_pure_latin and is_latin(character) is False:
target_pure_latin = False
return target_have_accents, target_pure_latin
def alphabet_languages(
characters: list[str], ignore_non_latin: bool = False
) -> list[str]:
"""
Return associated languages associated to given characters.
"""
languages: list[tuple[str, float]] = []
characters_set: frozenset[str] = frozenset(characters)
source_have_accents = any(is_accentuated(character) for character in characters)
for language, language_characters in FREQUENCIES.items():
target_have_accents, target_pure_latin = get_target_features(language)
if ignore_non_latin and target_pure_latin is False:
continue
if target_have_accents is False and source_have_accents:
continue
character_count: int = len(language_characters)
character_match_count: int = len(_FREQUENCIES_SET[language] & characters_set)
ratio: float = character_match_count / character_count
if ratio >= 0.2:
languages.append((language, ratio))
languages = sorted(languages, key=lambda x: x[1], reverse=True)
return [compatible_language[0] for compatible_language in languages]
def characters_popularity_compare(
language: str, ordered_characters: list[str]
) -> float:
"""
Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language.
The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit).
Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.)
"""
if language not in FREQUENCIES:
raise ValueError(f"{language} not available") # Defensive:
character_approved_count: int = 0
frequencies_language_set: frozenset[str] = _FREQUENCIES_SET[language]
lang_rank: dict[str, int] = _FREQUENCIES_RANK[language]
ordered_characters_count: int = len(ordered_characters)
target_language_characters_count: int = len(FREQUENCIES[language])
large_alphabet: bool = target_language_characters_count > 26
expected_projection_ratio: float = (
target_language_characters_count / ordered_characters_count
)
# Pre-built rank dict for ordered_characters (avoids repeated list slicing).
ordered_rank: dict[str, int] = {
char: rank for rank, char in enumerate(ordered_characters)
}
# Pre-compute characters common to both orderings.
# Avoids repeated `c in ordered_rank` dict lookups in the inner counts.
common_chars: list[tuple[int, int]] = [
(lr, ordered_rank[c]) for c, lr in lang_rank.items() if c in ordered_rank
]
# Pre-extract lr and orr arrays for faster iteration in the inner loop.
# Plain integer loops with local arrays are much faster under mypyc than
# generator expression sums over a list of tuples.
common_count: int = len(common_chars)
common_lr: list[int] = [p[0] for p in common_chars]
common_orr: list[int] = [p[1] for p in common_chars]
for character, character_rank in zip(
ordered_characters, range(0, ordered_characters_count)
):
if character not in frequencies_language_set:
continue
character_rank_in_language: int = lang_rank[character]
character_rank_projection: int = int(character_rank * expected_projection_ratio)
if (
large_alphabet is False
and abs(character_rank_projection - character_rank_in_language) > 4
):
continue
if (
large_alphabet is True
and abs(character_rank_projection - character_rank_in_language)
< target_language_characters_count / 3
):
character_approved_count += 1
continue
# Count how many characters appear "before" in both orderings,
# and how many appear "at or after" in both orderings.
# Single pass over pre-extracted arrays — much faster under mypyc
# than two generator expression sums.
before_match_count: int = 0
after_match_count: int = 0
for i in range(common_count):
lr_i: int = common_lr[i]
orr_i: int = common_orr[i]
if lr_i < character_rank_in_language:
if orr_i < character_rank:
before_match_count += 1
else:
if orr_i >= character_rank:
after_match_count += 1
after_len: int = target_language_characters_count - character_rank_in_language
if character_rank_in_language == 0 and before_match_count <= 4:
character_approved_count += 1
continue
if after_len == 0 and after_match_count <= 4:
character_approved_count += 1
continue
if (
character_rank_in_language > 0
and before_match_count / character_rank_in_language >= 0.4
) or (after_len > 0 and after_match_count / after_len >= 0.4):
character_approved_count += 1
continue
return character_approved_count / len(ordered_characters)
def alpha_unicode_split(decoded_sequence: str) -> list[str]:
"""
Given a decoded text sequence, return a list of str. Unicode range / alphabet separation.
Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list;
One containing the latin letters and the other hebrew.
"""
layers: dict[str, list[str]] = {}
# Fast path: track single-layer key to skip dict iteration for single-script text.
single_layer_key: str | None = None
multi_layer: bool = False
# Cache the last character_range and its resolved layer to avoid repeated
# is_suspiciously_successive_range calls for consecutive same-range chars.
prev_character_range: str | None = None
prev_layer_target: str | None = None
for character in decoded_sequence:
if character.isalpha() is False:
continue
# ASCII fast-path: a-z and A-Z are always "Basic Latin".
# Avoids unicode_range() function call overhead for the most common case.
character_ord: int = ord(character)
if character_ord < 128:
character_range: str | None = "Basic Latin"
else:
character_range = unicode_range(character)
if character_range is None:
continue
# Fast path: same range as previous character → reuse cached layer target.
if character_range == prev_character_range:
if prev_layer_target is not None:
layers[prev_layer_target].append(character)
continue
layer_target_range: str | None = None
if multi_layer:
for discovered_range in layers:
if (
is_suspiciously_successive_range(discovered_range, character_range)
is False
):
layer_target_range = discovered_range
break
elif single_layer_key is not None:
if (
is_suspiciously_successive_range(single_layer_key, character_range)
is False
):
layer_target_range = single_layer_key
if layer_target_range is None:
layer_target_range = character_range
if layer_target_range not in layers:
layers[layer_target_range] = []
if single_layer_key is None:
single_layer_key = layer_target_range
else:
multi_layer = True
layers[layer_target_range].append(character)
# Cache for next iteration
prev_character_range = character_range
prev_layer_target = layer_target_range
return ["".join(chars).lower() for chars in layers.values()]
def merge_coherence_ratios(results: list[CoherenceMatches]) -> CoherenceMatches:
"""
This function merge results previously given by the function coherence_ratio.
The return type is the same as coherence_ratio.
"""
per_language_ratios: dict[str, list[float]] = {}
for result in results:
for sub_result in result:
language, ratio = sub_result
if language not in per_language_ratios:
per_language_ratios[language] = [ratio]
continue
per_language_ratios[language].append(ratio)
merge = [
(
language,
round(
sum(per_language_ratios[language]) / len(per_language_ratios[language]),
4,
),
)
for language in per_language_ratios
]
return sorted(merge, key=lambda x: x[1], reverse=True)
def filter_alt_coherence_matches(results: CoherenceMatches) -> CoherenceMatches:
"""
We shall NOT return "English—" in CoherenceMatches because it is an alternative
of "English". This function only keeps the best match and remove the em-dash in it.
"""
index_results: dict[str, list[float]] = dict()
for result in results:
language, ratio = result
no_em_name: str = language.replace("—", "")
if no_em_name not in index_results:
index_results[no_em_name] = []
index_results[no_em_name].append(ratio)
if any(len(index_results[e]) > 1 for e in index_results):
filtered_results: CoherenceMatches = []
for language in index_results:
filtered_results.append((language, max(index_results[language])))
return filtered_results
return results
@lru_cache(maxsize=2048)
def coherence_ratio(
decoded_sequence: str, threshold: float = 0.1, lg_inclusion: str | None = None
) -> CoherenceMatches:
"""
Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers.
A layer = Character extraction by alphabets/ranges.
"""
results: list[tuple[str, float]] = []
ignore_non_latin: bool = False
sufficient_match_count: int = 0
lg_inclusion_list = lg_inclusion.split(",") if lg_inclusion is not None else []
if "Latin Based" in lg_inclusion_list:
ignore_non_latin = True
lg_inclusion_list.remove("Latin Based")
for layer in alpha_unicode_split(decoded_sequence):
sequence_frequencies: TypeCounter[str] = Counter(layer)
most_common = sequence_frequencies.most_common()
character_count: int = len(layer)
if character_count <= TOO_SMALL_SEQUENCE:
continue
popular_character_ordered: list[str] = [c for c, o in most_common]
for language in lg_inclusion_list or alphabet_languages(
popular_character_ordered, ignore_non_latin
):
ratio: float = characters_popularity_compare(
language, popular_character_ordered
)
if ratio < threshold:
continue
elif ratio >= 0.8:
sufficient_match_count += 1
results.append((language, round(ratio, 4)))
if sufficient_match_count >= 3:
break
return sorted(
filter_alt_coherence_matches(results), key=lambda x: x[1], reverse=True
)

8
.venv/lib/python3.12/site-packages/charset_normalizer/cli/__init__.py Normal file
View File

@@ -0,0 +1,8 @@
from __future__ import annotations
from .__main__ import cli_detect, query_yes_no
__all__ = (
"cli_detect",
"query_yes_no",
)

362
.venv/lib/python3.12/site-packages/charset_normalizer/cli/__main__.py Normal file
View File

@@ -0,0 +1,362 @@
from __future__ import annotations
import argparse
import sys
import typing
from json import dumps
from os.path import abspath, basename, dirname, join, realpath
from platform import python_version
from unicodedata import unidata_version
import charset_normalizer.md as md_module
from charset_normalizer import from_fp
from charset_normalizer.models import CliDetectionResult
from charset_normalizer.version import __version__
def query_yes_no(question: str, default: str = "yes") -> bool: # Defensive:
"""Ask a yes/no question via input() and return the answer as a bool."""
prompt = " [Y/n] " if default == "yes" else " [y/N] "
while True:
choice = input(question + prompt).strip().lower()
if not choice:
return default == "yes"
if choice in ("y", "yes"):
return True
if choice in ("n", "no"):
return False
print("Please respond with 'y' or 'n'.")
class FileType:
"""Factory for creating file object types
Instances of FileType are typically passed as type= arguments to the
ArgumentParser add_argument() method.
Keyword Arguments:
- mode -- A string indicating how the file is to be opened. Accepts the
same values as the builtin open() function.
- bufsize -- The file's desired buffer size. Accepts the same values as
the builtin open() function.
- encoding -- The file's encoding. Accepts the same values as the
builtin open() function.
- errors -- A string indicating how encoding and decoding errors are to
be handled. Accepts the same value as the builtin open() function.
Backported from CPython 3.12
"""
def __init__(
self,
mode: str = "r",
bufsize: int = -1,
encoding: str | None = None,
errors: str | None = None,
):
self._mode = mode
self._bufsize = bufsize
self._encoding = encoding
self._errors = errors
def __call__(self, string: str) -> typing.IO: # type: ignore[type-arg]
# the special argument "-" means sys.std{in,out}
if string == "-":
if "r" in self._mode:
return sys.stdin.buffer if "b" in self._mode else sys.stdin
elif any(c in self._mode for c in "wax"):
return sys.stdout.buffer if "b" in self._mode else sys.stdout
else:
msg = f'argument "-" with mode {self._mode}'
raise ValueError(msg)
# all other arguments are used as file names
try:
return open(string, self._mode, self._bufsize, self._encoding, self._errors)
except OSError as e:
message = f"can't open '{string}': {e}"
raise argparse.ArgumentTypeError(message)
def __repr__(self) -> str:
args = self._mode, self._bufsize
kwargs = [("encoding", self._encoding), ("errors", self._errors)]
args_str = ", ".join(
[repr(arg) for arg in args if arg != -1]
+ [f"{kw}={arg!r}" for kw, arg in kwargs if arg is not None]
)
return f"{type(self).__name__}({args_str})"
def cli_detect(argv: list[str] | None = None) -> int:
"""
CLI assistant using ARGV and ArgumentParser
:param argv:
:return: 0 if everything is fine, anything else equal trouble
"""
parser = argparse.ArgumentParser(
description="The Real First Universal Charset Detector. "
"Discover originating encoding used on text file. "
"Normalize text to unicode."
)
parser.add_argument(
"files", type=FileType("rb"), nargs="+", help="File(s) to be analysed"
)
parser.add_argument(
"-v",
"--verbose",
action="store_true",
default=False,
dest="verbose",
help="Display complementary information about file if any. "
"Stdout will contain logs about the detection process.",
)
parser.add_argument(
"-a",
"--with-alternative",
action="store_true",
default=False,
dest="alternatives",
help="Output complementary possibilities if any. Top-level JSON WILL be a list.",
)
parser.add_argument(
"-n",
"--normalize",
action="store_true",
default=False,
dest="normalize",
help="Permit to normalize input file. If not set, program does not write anything.",
)
parser.add_argument(
"-m",
"--minimal",
action="store_true",
default=False,
dest="minimal",
help="Only output the charset detected to STDOUT. Disabling JSON output.",
)
parser.add_argument(
"-r",
"--replace",
action="store_true",
default=False,
dest="replace",
help="Replace file when trying to normalize it instead of creating a new one.",
)
parser.add_argument(
"-f",
"--force",
action="store_true",
default=False,
dest="force",
help="Replace file without asking if you are sure, use this flag with caution.",
)
parser.add_argument(
"-i",
"--no-preemptive",
action="store_true",
default=False,
dest="no_preemptive",
help="Disable looking at a charset declaration to hint the detector.",
)
parser.add_argument(
"-t",
"--threshold",
action="store",
default=0.2,
type=float,
dest="threshold",
help="Define a custom maximum amount of noise allowed in decoded content. 0. <= noise <= 1.",
)
parser.add_argument(
"--version",
action="version",
version="Charset-Normalizer {} - Python {} - Unicode {} - SpeedUp {}".format(
__version__,
python_version(),
unidata_version,
"OFF" if md_module.__file__.lower().endswith(".py") else "ON",
),
help="Show version information and exit.",
)
args = parser.parse_args(argv)
if args.replace is True and args.normalize is False:
if args.files:
for my_file in args.files:
my_file.close()
print("Use --replace in addition of --normalize only.", file=sys.stderr)
return 1
if args.force is True and args.replace is False:
if args.files:
for my_file in args.files:
my_file.close()
print("Use --force in addition of --replace only.", file=sys.stderr)
return 1
if args.threshold < 0.0 or args.threshold > 1.0:
if args.files:
for my_file in args.files:
my_file.close()
print("--threshold VALUE should be between 0. AND 1.", file=sys.stderr)
return 1
x_ = []
for my_file in args.files:
matches = from_fp(
my_file,
threshold=args.threshold,
explain=args.verbose,
preemptive_behaviour=args.no_preemptive is False,
)
best_guess = matches.best()
if best_guess is None:
print(
'Unable to identify originating encoding for "{}". {}'.format(
my_file.name,
(
"Maybe try increasing maximum amount of chaos."
if args.threshold < 1.0
else ""
),
),
file=sys.stderr,
)
x_.append(
CliDetectionResult(
abspath(my_file.name),
None,
[],
[],
"Unknown",
[],
False,
1.0,
0.0,
None,
True,
)
)
else:
cli_result = CliDetectionResult(
abspath(my_file.name),
best_guess.encoding,
best_guess.encoding_aliases,
[
cp
for cp in best_guess.could_be_from_charset
if cp != best_guess.encoding
],
best_guess.language,
best_guess.alphabets,
best_guess.bom,
best_guess.percent_chaos,
best_guess.percent_coherence,
None,
True,
)
x_.append(cli_result)
if len(matches) > 1 and args.alternatives:
for el in matches:
if el != best_guess:
x_.append(
CliDetectionResult(
abspath(my_file.name),
el.encoding,
el.encoding_aliases,
[
cp
for cp in el.could_be_from_charset
if cp != el.encoding
],
el.language,
el.alphabets,
el.bom,
el.percent_chaos,
el.percent_coherence,
None,
False,
)
)
if args.normalize is True:
if best_guess.encoding.startswith("utf") is True:
print(
'"{}" file does not need to be normalized, as it already came from unicode.'.format(
my_file.name
),
file=sys.stderr,
)
if my_file.closed is False:
my_file.close()
continue
dir_path = dirname(realpath(my_file.name))
file_name = basename(realpath(my_file.name))
o_: list[str] = file_name.split(".")
if args.replace is False:
o_.insert(-1, best_guess.encoding)
if my_file.closed is False:
my_file.close()
elif (
args.force is False
and query_yes_no(
'Are you sure to normalize "{}" by replacing it ?'.format(
my_file.name
),
"no",
)
is False
):
if my_file.closed is False:
my_file.close()
continue
try:
cli_result.unicode_path = join(dir_path, ".".join(o_))
with open(cli_result.unicode_path, "wb") as fp:
fp.write(best_guess.output())
except OSError as e: # Defensive:
print(str(e), file=sys.stderr)
if my_file.closed is False:
my_file.close()
return 2
if my_file.closed is False:
my_file.close()
if args.minimal is False:
print(
dumps(
[el.__dict__ for el in x_] if len(x_) > 1 else x_[0].__dict__,
ensure_ascii=True,
indent=4,
)
)
else:
for my_file in args.files:
print(
", ".join(
[
el.encoding or "undefined"
for el in x_
if el.path == abspath(my_file.name)
]
)
)
return 0
if __name__ == "__main__": # Defensive:
cli_detect()

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/cli/__pycache__/__init__.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/cli/__pycache__/__main__.cpython-312.pyc Normal file
View File

Binary file not shown.

2050
.venv/lib/python3.12/site-packages/charset_normalizer/constant.py Normal file
View File

File diff suppressed because it is too large Load Diff

79
.venv/lib/python3.12/site-packages/charset_normalizer/legacy.py Normal file
View File

@@ -0,0 +1,79 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from warnings import warn
from .api import from_bytes
from .constant import CHARDET_CORRESPONDENCE, TOO_SMALL_SEQUENCE
if TYPE_CHECKING:
from typing import TypedDict
class ResultDict(TypedDict):
encoding: str | None
language: str
confidence: float | None
def detect(
byte_str: bytes, should_rename_legacy: bool = False, **kwargs: Any
) -> ResultDict:
"""
chardet legacy method
Detect the encoding of the given byte string. It should be mostly backward-compatible.
Encoding name will match Chardet own writing whenever possible. (Not on encoding name unsupported by it)
This function is deprecated and should be used to migrate your project easily, consult the documentation for
further information. Not planned for removal.
:param byte_str: The byte sequence to examine.
:param should_rename_legacy: Should we rename legacy encodings
to their more modern equivalents?
"""
if len(kwargs):
warn(
f"charset-normalizer disregard arguments '{','.join(list(kwargs.keys()))}' in legacy function detect()"
)
if not isinstance(byte_str, (bytearray, bytes)):
raise TypeError( # pragma: nocover
f"Expected object of type bytes or bytearray, got: {type(byte_str)}"
)
if isinstance(byte_str, bytearray):
byte_str = bytes(byte_str)
r = from_bytes(byte_str).best()
encoding = r.encoding if r is not None else None
language = r.language if r is not None and r.language != "Unknown" else ""
confidence = 1.0 - r.chaos if r is not None else None
# automatically lower confidence
# on small bytes samples.
# https://github.com/jawah/charset_normalizer/issues/391
if (
confidence is not None
and confidence >= 0.9
and encoding
not in {
"utf_8",
"ascii",
}
and r.bom is False # type: ignore[union-attr]
and len(byte_str) < TOO_SMALL_SEQUENCE
):
confidence -= 0.2
# Note: CharsetNormalizer does not return 'UTF-8-SIG' as the sig get stripped in the detection/normalization process
# but chardet does return 'utf-8-sig' and it is a valid codec name.
if r is not None and encoding == "utf_8" and r.bom:
encoding += "_sig"
if should_rename_legacy is False and encoding in CHARDET_CORRESPONDENCE:
encoding = CHARDET_CORRESPONDENCE[encoding]
return {
"encoding": encoding,
"language": language,
"confidence": confidence,
}

BIN
.venv/lib/python3.12/site-packages/charset_normalizer/md.cpython-312-aarch64-linux-gnu.so Executable file
View File

Binary file not shown.

936
.venv/lib/python3.12/site-packages/charset_normalizer/md.py Normal file
View File

@@ -0,0 +1,936 @@
from __future__ import annotations
import sys
from functools import lru_cache
from logging import getLogger
if sys.version_info >= (3, 8):
from typing import final
else:
try:
from typing_extensions import final
except ImportError:
def final(cls): # type: ignore[misc,no-untyped-def]
return cls
from .constant import (
COMMON_CJK_CHARACTERS,
COMMON_SAFE_ASCII_CHARACTERS,
TRACE,
UNICODE_SECONDARY_RANGE_KEYWORD,
_ACCENTUATED,
_ARABIC,
_ARABIC_ISOLATED_FORM,
_CJK,
_HANGUL,
_HIRAGANA,
_KATAKANA,
_LATIN,
_THAI,
)
from .utils import (
_character_flags,
is_emoticon,
is_punctuation,
is_separator,
is_symbol,
remove_accent,
unicode_range,
)
# Combined bitmask for CJK/Hangul/Katakana/Hiragana/Thai glyph detection.
_GLYPH_MASK: int = _CJK | _HANGUL | _KATAKANA | _HIRAGANA | _THAI
@final
class CharInfo:
"""Pre-computed character properties shared across all detectors.
Instantiated once and reused via :meth:`update` on every character
in the hot loop so that redundant calls to str methods
(``isalpha``, ``isupper``, …) and cached utility functions
(``_character_flags``, ``is_punctuation``, …) are avoided when
several plugins need the same information.
"""
__slots__ = (
"character",
"printable",
"alpha",
"upper",
"lower",
"space",
"digit",
"is_ascii",
"case_variable",
"flags",
"accentuated",
"latin",
"is_cjk",
"is_arabic",
"is_glyph",
"punct",
"sym",
)
def __init__(self) -> None:
self.character: str = ""
self.printable: bool = False
self.alpha: bool = False
self.upper: bool = False
self.lower: bool = False
self.space: bool = False
self.digit: bool = False
self.is_ascii: bool = False
self.case_variable: bool = False
self.flags: int = 0
self.accentuated: bool = False
self.latin: bool = False
self.is_cjk: bool = False
self.is_arabic: bool = False
self.is_glyph: bool = False
self.punct: bool = False
self.sym: bool = False
def update(self, character: str) -> None:
"""Update all properties for *character* (called once per character)."""
self.character = character
# ASCII fast-path: for characters with ord < 128, we can skip
# _character_flags() entirely and derive most properties from ord.
o: int = ord(character)
if o < 128:
self.is_ascii = True
self.accentuated = False
self.is_cjk = False
self.is_arabic = False
self.is_glyph = False
# ASCII alpha: a-z (97-122) or A-Z (65-90)
if 65 <= o <= 90:
# Uppercase ASCII letter
self.alpha = True
self.upper = True
self.lower = False
self.space = False
self.digit = False
self.printable = True
self.case_variable = True
self.flags = _LATIN
self.latin = True
self.punct = False
self.sym = False
elif 97 <= o <= 122:
# Lowercase ASCII letter
self.alpha = True
self.upper = False
self.lower = True
self.space = False
self.digit = False
self.printable = True
self.case_variable = True
self.flags = _LATIN
self.latin = True
self.punct = False
self.sym = False
elif 48 <= o <= 57:
# ASCII digit 0-9
self.alpha = False
self.upper = False
self.lower = False
self.space = False
self.digit = True
self.printable = True
self.case_variable = False
self.flags = 0
self.latin = False
self.punct = False
self.sym = False
elif o == 32 or (9 <= o <= 13):
# Space, tab, newline, etc.
self.alpha = False
self.upper = False
self.lower = False
self.space = True
self.digit = False
self.printable = o == 32
self.case_variable = False
self.flags = 0
self.latin = False
self.punct = False
self.sym = False
else:
# Other ASCII (punctuation, symbols, control chars)
self.printable = character.isprintable()
self.alpha = False
self.upper = False
self.lower = False
self.space = False
self.digit = False
self.case_variable = False
self.flags = 0
self.latin = False
self.punct = is_punctuation(character) if self.printable else False
self.sym = is_symbol(character) if self.printable else False
else:
# Non-ASCII path
self.is_ascii = False
self.printable = character.isprintable()
self.alpha = character.isalpha()
self.upper = character.isupper()
self.lower = character.islower()
self.space = character.isspace()
self.digit = character.isdigit()
self.case_variable = self.lower != self.upper
# Flag-based classification (single unicodedata.name() call, lru-cached)
flags: int
if self.alpha:
flags = _character_flags(character)
else:
flags = 0
self.flags = flags
self.accentuated = bool(flags & _ACCENTUATED)
self.latin = bool(flags & _LATIN)
self.is_cjk = bool(flags & _CJK)
self.is_arabic = bool(flags & _ARABIC)
self.is_glyph = bool(flags & _GLYPH_MASK)
# Eagerly compute punct and sym (avoids property dispatch overhead
# on 300K+ accesses in the hot loop).
self.punct = is_punctuation(character) if self.printable else False
self.sym = is_symbol(character) if self.printable else False
class MessDetectorPlugin:
"""
Base abstract class used for mess detection plugins.
All detectors MUST extend and implement given methods.
"""
__slots__ = ()
def feed_info(self, character: str, info: CharInfo) -> None:
"""
The main routine to be executed upon character.
Insert the logic in witch the text would be considered chaotic.
"""
raise NotImplementedError # Defensive:
def reset(self) -> None: # Defensive:
"""
Permit to reset the plugin to the initial state.
"""
raise NotImplementedError
@property
def ratio(self) -> float:
"""
Compute the chaos ratio based on what your feed() has seen.
Must NOT be lower than 0.; No restriction gt 0.
"""
raise NotImplementedError # Defensive:
@final
class TooManySymbolOrPunctuationPlugin(MessDetectorPlugin):
__slots__ = (
"_punctuation_count",
"_symbol_count",
"_character_count",
"_last_printable_char",
"_frenzy_symbol_in_word",
)
def __init__(self) -> None:
self._punctuation_count: int = 0
self._symbol_count: int = 0
self._character_count: int = 0
self._last_printable_char: str | None = None
self._frenzy_symbol_in_word: bool = False
def feed_info(self, character: str, info: CharInfo) -> None:
"""Optimized feed using pre-computed character info."""
self._character_count += 1
if (
character != self._last_printable_char
and character not in COMMON_SAFE_ASCII_CHARACTERS
):
if info.punct:
self._punctuation_count += 1
elif not info.digit and info.sym and not is_emoticon(character):
self._symbol_count += 2
self._last_printable_char = character
def reset(self) -> None: # Abstract
self._punctuation_count = 0
self._character_count = 0
self._symbol_count = 0
@property
def ratio(self) -> float:
if self._character_count == 0:
return 0.0
ratio_of_punctuation: float = (
self._punctuation_count + self._symbol_count
) / self._character_count
return ratio_of_punctuation if ratio_of_punctuation >= 0.3 else 0.0
@final
class TooManyAccentuatedPlugin(MessDetectorPlugin):
__slots__ = ("_character_count", "_accentuated_count")
def __init__(self) -> None:
self._character_count: int = 0
self._accentuated_count: int = 0
def feed_info(self, character: str, info: CharInfo) -> None:
"""Optimized feed using pre-computed character info."""
self._character_count += 1
if info.accentuated:
self._accentuated_count += 1
def reset(self) -> None: # Abstract
self._character_count = 0
self._accentuated_count = 0
@property
def ratio(self) -> float:
if self._character_count < 8:
return 0.0
ratio_of_accentuation: float = self._accentuated_count / self._character_count
return ratio_of_accentuation if ratio_of_accentuation >= 0.35 else 0.0
@final
class UnprintablePlugin(MessDetectorPlugin):
__slots__ = ("_unprintable_count", "_character_count")
def __init__(self) -> None:
self._unprintable_count: int = 0
self._character_count: int = 0
def feed_info(self, character: str, info: CharInfo) -> None:
"""Optimized feed using pre-computed character info."""
if (
not info.space
and not info.printable
and character != "\x1a"
and character != "\ufeff"
):
self._unprintable_count += 1
self._character_count += 1
def reset(self) -> None: # Abstract
self._unprintable_count = 0
@property
def ratio(self) -> float:
if self._character_count == 0: # Defensive:
return 0.0
return (self._unprintable_count * 8) / self._character_count
@final
class SuspiciousDuplicateAccentPlugin(MessDetectorPlugin):
__slots__ = (
"_successive_count",
"_character_count",
"_last_latin_character",
"_last_was_accentuated",
)
def __init__(self) -> None:
self._successive_count: int = 0
self._character_count: int = 0
self._last_latin_character: str | None = None
self._last_was_accentuated: bool = False
def feed_info(self, character: str, info: CharInfo) -> None:
"""Optimized feed using pre-computed character info."""
self._character_count += 1
if (
self._last_latin_character is not None
and info.accentuated
and self._last_was_accentuated
):
if info.upper and self._last_latin_character.isupper():
self._successive_count += 1
if remove_accent(character) == remove_accent(self._last_latin_character):
self._successive_count += 1
self._last_latin_character = character
self._last_was_accentuated = info.accentuated
def reset(self) -> None: # Abstract
self._successive_count = 0
self._character_count = 0
self._last_latin_character = None
self._last_was_accentuated = False
@property
def ratio(self) -> float:
if self._character_count == 0:
return 0.0
return (self._successive_count * 2) / self._character_count
@final
class SuspiciousRange(MessDetectorPlugin):
__slots__ = (
"_suspicious_successive_range_count",
"_character_count",
"_last_printable_seen",
"_last_printable_range",
)
def __init__(self) -> None:
self._suspicious_successive_range_count: int = 0
self._character_count: int = 0
self._last_printable_seen: str | None = None
self._last_printable_range: str | None = None
def feed_info(self, character: str, info: CharInfo) -> None:
"""Optimized feed using pre-computed character info."""
self._character_count += 1
if info.space or info.punct or character in COMMON_SAFE_ASCII_CHARACTERS:
self._last_printable_seen = None
self._last_printable_range = None
return
if self._last_printable_seen is None:
self._last_printable_seen = character
self._last_printable_range = unicode_range(character)
return
unicode_range_a: str | None = self._last_printable_range
unicode_range_b: str | None = unicode_range(character)
if is_suspiciously_successive_range(unicode_range_a, unicode_range_b):
self._suspicious_successive_range_count += 1
self._last_printable_seen = character
self._last_printable_range = unicode_range_b
def reset(self) -> None: # Abstract
self._character_count = 0
self._suspicious_successive_range_count = 0
self._last_printable_seen = None
self._last_printable_range = None
@property
def ratio(self) -> float:
if self._character_count <= 13:
return 0.0
ratio_of_suspicious_range_usage: float = (
self._suspicious_successive_range_count * 2
) / self._character_count
return ratio_of_suspicious_range_usage
@final
class SuperWeirdWordPlugin(MessDetectorPlugin):
__slots__ = (
"_word_count",
"_bad_word_count",
"_foreign_long_count",
"_is_current_word_bad",
"_foreign_long_watch",
"_character_count",
"_bad_character_count",
"_buffer_length",
"_buffer_last_char",
"_buffer_last_char_accentuated",
"_buffer_accent_count",
"_buffer_glyph_count",
"_buffer_upper_count",
)
def __init__(self) -> None:
self._word_count: int = 0
self._bad_word_count: int = 0
self._foreign_long_count: int = 0
self._is_current_word_bad: bool = False
self._foreign_long_watch: bool = False
self._character_count: int = 0
self._bad_character_count: int = 0
self._buffer_length: int = 0
self._buffer_last_char: str | None = None
self._buffer_last_char_accentuated: bool = False
self._buffer_accent_count: int = 0
self._buffer_glyph_count: int = 0
self._buffer_upper_count: int = 0
def feed_info(self, character: str, info: CharInfo) -> None:
"""Optimized feed using pre-computed character info."""
if info.alpha:
self._buffer_length += 1
self._buffer_last_char = character
if info.upper:
self._buffer_upper_count += 1
self._buffer_last_char_accentuated = info.accentuated
if info.accentuated:
self._buffer_accent_count += 1
if (
not self._foreign_long_watch
and (not info.latin or info.accentuated)
and not info.is_glyph
):
self._foreign_long_watch = True
if info.is_glyph:
self._buffer_glyph_count += 1
return
if not self._buffer_length:
return
if info.space or info.punct or is_separator(character):
self._word_count += 1
buffer_length: int = self._buffer_length
self._character_count += buffer_length
if buffer_length >= 4:
if self._buffer_accent_count / buffer_length >= 0.5:
self._is_current_word_bad = True
elif (
self._buffer_last_char_accentuated
and self._buffer_last_char.isupper() # type: ignore[union-attr]
and self._buffer_upper_count != buffer_length
):
self._foreign_long_count += 1
self._is_current_word_bad = True
elif self._buffer_glyph_count == 1:
self._is_current_word_bad = True
self._foreign_long_count += 1
if buffer_length >= 24 and self._foreign_long_watch:
probable_camel_cased: bool = (
self._buffer_upper_count > 0
and self._buffer_upper_count / buffer_length <= 0.3
)
if not probable_camel_cased:
self._foreign_long_count += 1
self._is_current_word_bad = True
if self._is_current_word_bad:
self._bad_word_count += 1
self._bad_character_count += buffer_length
self._is_current_word_bad = False
self._foreign_long_watch = False
self._buffer_length = 0
self._buffer_last_char = None
self._buffer_last_char_accentuated = False
self._buffer_accent_count = 0
self._buffer_glyph_count = 0
self._buffer_upper_count = 0
elif (
character not in {"<", ">", "-", "=", "~", "|", "_"}
and not info.digit
and info.sym
):
self._is_current_word_bad = True
self._buffer_length += 1
self._buffer_last_char = character
self._buffer_last_char_accentuated = False
def reset(self) -> None: # Abstract
self._buffer_length = 0
self._buffer_last_char = None
self._buffer_last_char_accentuated = False
self._is_current_word_bad = False
self._foreign_long_watch = False
self._bad_word_count = 0
self._word_count = 0
self._character_count = 0
self._bad_character_count = 0
self._foreign_long_count = 0
self._buffer_accent_count = 0
self._buffer_glyph_count = 0
self._buffer_upper_count = 0
@property
def ratio(self) -> float:
if self._word_count <= 10 and self._foreign_long_count == 0:
return 0.0
return self._bad_character_count / self._character_count
@final
class CjkUncommonPlugin(MessDetectorPlugin):
"""
Detect messy CJK text that probably means nothing.
"""
__slots__ = ("_character_count", "_uncommon_count")
def __init__(self) -> None:
self._character_count: int = 0
self._uncommon_count: int = 0
def feed_info(self, character: str, info: CharInfo) -> None:
"""Optimized feed using pre-computed character info."""
self._character_count += 1
if character not in COMMON_CJK_CHARACTERS:
self._uncommon_count += 1
def reset(self) -> None: # Abstract
self._character_count = 0
self._uncommon_count = 0
@property
def ratio(self) -> float:
if self._character_count < 8:
return 0.0
uncommon_form_usage: float = self._uncommon_count / self._character_count
# we can be pretty sure it's garbage when uncommon characters are widely
# used. otherwise it could just be traditional chinese for example.
return uncommon_form_usage / 10 if uncommon_form_usage > 0.5 else 0.0
@final
class ArchaicUpperLowerPlugin(MessDetectorPlugin):
__slots__ = (
"_buf",
"_character_count_since_last_sep",
"_successive_upper_lower_count",
"_successive_upper_lower_count_final",
"_character_count",
"_last_alpha_seen",
"_last_alpha_seen_upper",
"_last_alpha_seen_lower",
"_current_ascii_only",
)
def __init__(self) -> None:
self._buf: bool = False
self._character_count_since_last_sep: int = 0
self._successive_upper_lower_count: int = 0
self._successive_upper_lower_count_final: int = 0
self._character_count: int = 0
self._last_alpha_seen: str | None = None
self._last_alpha_seen_upper: bool = False
self._last_alpha_seen_lower: bool = False
self._current_ascii_only: bool = True
def feed_info(self, character: str, info: CharInfo) -> None:
"""Optimized feed using pre-computed character info."""
is_concerned: bool = info.alpha and info.case_variable
chunk_sep: bool = not is_concerned
if chunk_sep and self._character_count_since_last_sep > 0:
if (
self._character_count_since_last_sep <= 64
and not info.digit
and not self._current_ascii_only
):
self._successive_upper_lower_count_final += (
self._successive_upper_lower_count
)
self._successive_upper_lower_count = 0
self._character_count_since_last_sep = 0
self._last_alpha_seen = None
self._buf = False
self._character_count += 1
self._current_ascii_only = True
return
if self._current_ascii_only and not info.is_ascii:
self._current_ascii_only = False
if self._last_alpha_seen is not None:
if (info.upper and self._last_alpha_seen_lower) or (
info.lower and self._last_alpha_seen_upper
):
if self._buf:
self._successive_upper_lower_count += 2
self._buf = False
else:
self._buf = True
else:
self._buf = False
self._character_count += 1
self._character_count_since_last_sep += 1
self._last_alpha_seen = character
self._last_alpha_seen_upper = info.upper
self._last_alpha_seen_lower = info.lower
def reset(self) -> None: # Abstract
self._character_count = 0
self._character_count_since_last_sep = 0
self._successive_upper_lower_count = 0
self._successive_upper_lower_count_final = 0
self._last_alpha_seen = None
self._last_alpha_seen_upper = False
self._last_alpha_seen_lower = False
self._buf = False
self._current_ascii_only = True
@property
def ratio(self) -> float:
if self._character_count == 0: # Defensive:
return 0.0
return self._successive_upper_lower_count_final / self._character_count
@final
class ArabicIsolatedFormPlugin(MessDetectorPlugin):
__slots__ = ("_character_count", "_isolated_form_count")
def __init__(self) -> None:
self._character_count: int = 0
self._isolated_form_count: int = 0
def reset(self) -> None: # Abstract
self._character_count = 0
self._isolated_form_count = 0
def feed_info(self, character: str, info: CharInfo) -> None:
"""Optimized feed using pre-computed character info."""
self._character_count += 1
if info.flags & _ARABIC_ISOLATED_FORM:
self._isolated_form_count += 1
@property
def ratio(self) -> float:
if self._character_count < 8:
return 0.0
isolated_form_usage: float = self._isolated_form_count / self._character_count
return isolated_form_usage
@lru_cache(maxsize=1024)
def is_suspiciously_successive_range(
unicode_range_a: str | None, unicode_range_b: str | None
) -> bool:
"""
Determine if two Unicode range seen next to each other can be considered as suspicious.
"""
if unicode_range_a is None or unicode_range_b is None:
return True
if unicode_range_a == unicode_range_b:
return False
if "Latin" in unicode_range_a and "Latin" in unicode_range_b:
return False
if "Emoticons" in unicode_range_a or "Emoticons" in unicode_range_b:
return False
# Latin characters can be accompanied with a combining diacritical mark
# eg. Vietnamese.
if ("Latin" in unicode_range_a or "Latin" in unicode_range_b) and (
"Combining" in unicode_range_a or "Combining" in unicode_range_b
):
return False
keywords_range_a, keywords_range_b = (
unicode_range_a.split(" "),
unicode_range_b.split(" "),
)
for el in keywords_range_a:
if el in UNICODE_SECONDARY_RANGE_KEYWORD:
continue
if el in keywords_range_b:
return False
# Japanese Exception
range_a_jp_chars, range_b_jp_chars = (
unicode_range_a
in (
"Hiragana",
"Katakana",
),
unicode_range_b in ("Hiragana", "Katakana"),
)
if (range_a_jp_chars or range_b_jp_chars) and (
"CJK" in unicode_range_a or "CJK" in unicode_range_b
):
return False
if range_a_jp_chars and range_b_jp_chars:
return False
if "Hangul" in unicode_range_a or "Hangul" in unicode_range_b:
if "CJK" in unicode_range_a or "CJK" in unicode_range_b:
return False
if unicode_range_a == "Basic Latin" or unicode_range_b == "Basic Latin":
return False
# Chinese/Japanese use dedicated range for punctuation and/or separators.
if ("CJK" in unicode_range_a or "CJK" in unicode_range_b) or (
unicode_range_a in ["Katakana", "Hiragana"]
and unicode_range_b in ["Katakana", "Hiragana"]
):
if "Punctuation" in unicode_range_a or "Punctuation" in unicode_range_b:
return False
if "Forms" in unicode_range_a or "Forms" in unicode_range_b:
return False
if unicode_range_a == "Basic Latin" or unicode_range_b == "Basic Latin":
return False
return True
@lru_cache(maxsize=2048)
def mess_ratio(
decoded_sequence: str, maximum_threshold: float = 0.2, debug: bool = False
) -> float:
"""
Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier.
"""
seq_len: int = len(decoded_sequence)
if seq_len < 511:
step: int = 32
elif seq_len < 1024:
step = 64
else:
step = 128
# Create each detector as a named local variable (unrolled from the generic loop).
# This eliminates per-character iteration over the detector list and
# per-character eligible() virtual dispatch, while keeping every plugin class
# intact and fully readable.
d_sp: TooManySymbolOrPunctuationPlugin = TooManySymbolOrPunctuationPlugin()
d_ta: TooManyAccentuatedPlugin = TooManyAccentuatedPlugin()
d_up: UnprintablePlugin = UnprintablePlugin()
d_sda: SuspiciousDuplicateAccentPlugin = SuspiciousDuplicateAccentPlugin()
d_sr: SuspiciousRange = SuspiciousRange()
d_sw: SuperWeirdWordPlugin = SuperWeirdWordPlugin()
d_cu: CjkUncommonPlugin = CjkUncommonPlugin()
d_au: ArchaicUpperLowerPlugin = ArchaicUpperLowerPlugin()
d_ai: ArabicIsolatedFormPlugin = ArabicIsolatedFormPlugin()
# Local references for feed_info methods called in the hot loop.
d_sp_feed = d_sp.feed_info
d_ta_feed = d_ta.feed_info
d_up_feed = d_up.feed_info
d_sda_feed = d_sda.feed_info
d_sr_feed = d_sr.feed_info
d_sw_feed = d_sw.feed_info
d_cu_feed = d_cu.feed_info
d_au_feed = d_au.feed_info
d_ai_feed = d_ai.feed_info
# Single reusable CharInfo object (avoids per-character allocation).
info: CharInfo = CharInfo()
info_update = info.update
mean_mess_ratio: float
for block_start in range(0, seq_len, step):
for character in decoded_sequence[block_start : block_start + step]:
# Pre-compute all character properties once (shared across all plugins).
info_update(character)
# Detectors with eligible() == always True
d_up_feed(character, info)
d_sw_feed(character, info)
d_au_feed(character, info)
# Detectors with eligible() == isprintable
if info.printable:
d_sp_feed(character, info)
d_sr_feed(character, info)
# Detectors with eligible() == isalpha
if info.alpha:
d_ta_feed(character, info)
# SuspiciousDuplicateAccent: isalpha() and is_latin()
if info.latin:
d_sda_feed(character, info)
# CjkUncommon: is_cjk()
if info.is_cjk:
d_cu_feed(character, info)
# ArabicIsolatedForm: is_arabic()
if info.is_arabic:
d_ai_feed(character, info)
mean_mess_ratio = (
d_sp.ratio
+ d_ta.ratio
+ d_up.ratio
+ d_sda.ratio
+ d_sr.ratio
+ d_sw.ratio
+ d_cu.ratio
+ d_au.ratio
+ d_ai.ratio
)
if mean_mess_ratio >= maximum_threshold:
break
else:
# Flush last word buffer in SuperWeirdWordPlugin via trailing newline.
info_update("\n")
d_sw_feed("\n", info)
d_au_feed("\n", info)
d_up_feed("\n", info)
mean_mess_ratio = (
d_sp.ratio
+ d_ta.ratio
+ d_up.ratio
+ d_sda.ratio
+ d_sr.ratio
+ d_sw.ratio
+ d_cu.ratio
+ d_au.ratio
+ d_ai.ratio
)
if debug: # Defensive:
logger = getLogger("charset_normalizer")
logger.log(
TRACE,
"Mess-detector extended-analysis start. "
f"intermediary_mean_mess_ratio_calc={step} mean_mess_ratio={mean_mess_ratio} "
f"maximum_threshold={maximum_threshold}",
)
if seq_len > 16:
logger.log(TRACE, f"Starting with: {decoded_sequence[:16]}")
logger.log(TRACE, f"Ending with: {decoded_sequence[-16::]}")
for dt in [d_sp, d_ta, d_up, d_sda, d_sr, d_sw, d_cu, d_au, d_ai]:
logger.log(TRACE, f"{dt.__class__}: {dt.ratio}")
return round(mean_mess_ratio, 3)

369
.venv/lib/python3.12/site-packages/charset_normalizer/models.py Normal file
View File

@@ -0,0 +1,369 @@
from __future__ import annotations
from encodings.aliases import aliases
from json import dumps
from re import sub
from typing import Any, Iterator, List, Tuple
from .constant import RE_POSSIBLE_ENCODING_INDICATION, TOO_BIG_SEQUENCE
from .utils import iana_name, is_multi_byte_encoding, unicode_range
class CharsetMatch:
def __init__(
self,
payload: bytes | bytearray,
guessed_encoding: str,
mean_mess_ratio: float,
has_sig_or_bom: bool,
languages: CoherenceMatches,
decoded_payload: str | None = None,
preemptive_declaration: str | None = None,
):
self._payload: bytes | bytearray = payload
self._encoding: str = guessed_encoding
self._mean_mess_ratio: float = mean_mess_ratio
self._languages: CoherenceMatches = languages
self._has_sig_or_bom: bool = has_sig_or_bom
self._unicode_ranges: list[str] | None = None
self._leaves: list[CharsetMatch] = []
self._mean_coherence_ratio: float = 0.0
self._output_payload: bytes | None = None
self._output_encoding: str | None = None
self._string: str | None = decoded_payload
self._preemptive_declaration: str | None = preemptive_declaration
def __eq__(self, other: object) -> bool:
if not isinstance(other, CharsetMatch):
if isinstance(other, str):
return iana_name(other) == self.encoding
return False
return self.encoding == other.encoding and self.fingerprint == other.fingerprint
def __lt__(self, other: object) -> bool:
"""
Implemented to make sorted available upon CharsetMatches items.
"""
if not isinstance(other, CharsetMatch):
raise ValueError
chaos_difference: float = abs(self.chaos - other.chaos)
coherence_difference: float = abs(self.coherence - other.coherence)
# Below 0.5% difference --> Use Coherence
if chaos_difference < 0.005 and coherence_difference > 0.02:
return self.coherence > other.coherence
elif chaos_difference < 0.005 and coherence_difference <= 0.02:
# When having a difficult decision, use the result that decoded as many multi-byte as possible.
# preserve RAM usage!
if len(self._payload) >= TOO_BIG_SEQUENCE:
return self.chaos < other.chaos
return self.multi_byte_usage > other.multi_byte_usage
return self.chaos < other.chaos
@property
def multi_byte_usage(self) -> float:
return 1.0 - (len(str(self)) / len(self.raw))
def __str__(self) -> str:
# Lazy Str Loading
if self._string is None:
self._string = str(self._payload, self._encoding, "strict")
# UTF-7 BOM is encoded in modified Base64 whose byte boundary
# can overlap with the next character, so raw-byte stripping
# is unreliable. Strip the decoded BOM character instead.
if (
self._has_sig_or_bom
and self._encoding == "utf_7"
and self._string
and self._string[0] == "\ufeff"
):
self._string = self._string[1:]
return self._string
def __repr__(self) -> str:
return f"<CharsetMatch '{self.encoding}' fp({self.fingerprint})>"
def add_submatch(self, other: CharsetMatch) -> None:
if not isinstance(other, CharsetMatch) or other == self:
raise ValueError(
"Unable to add instance <{}> as a submatch of a CharsetMatch".format(
other.__class__
)
)
other._string = None # Unload RAM usage; dirty trick.
self._leaves.append(other)
@property
def encoding(self) -> str:
return self._encoding
@property
def encoding_aliases(self) -> list[str]:
"""
Encoding name are known by many name, using this could help when searching for IBM855 when it's listed as CP855.
"""
also_known_as: list[str] = []
for u, p in aliases.items():
if self.encoding == u:
also_known_as.append(p)
elif self.encoding == p:
also_known_as.append(u)
return also_known_as
@property
def bom(self) -> bool:
return self._has_sig_or_bom
@property
def byte_order_mark(self) -> bool:
return self._has_sig_or_bom
@property
def languages(self) -> list[str]:
"""
Return the complete list of possible languages found in decoded sequence.
Usually not really useful. Returned list may be empty even if 'language' property return something != 'Unknown'.
"""
return [e[0] for e in self._languages]
@property
def language(self) -> str:
"""
Most probable language found in decoded sequence. If none were detected or inferred, the property will return
"Unknown".
"""
if not self._languages:
# Trying to infer the language based on the given encoding
# Its either English or we should not pronounce ourselves in certain cases.
if "ascii" in self.could_be_from_charset:
return "English"
# doing it there to avoid circular import
from charset_normalizer.cd import encoding_languages, mb_encoding_languages
languages = (
mb_encoding_languages(self.encoding)
if is_multi_byte_encoding(self.encoding)
else encoding_languages(self.encoding)
)
if len(languages) == 0 or "Latin Based" in languages:
return "Unknown"
return languages[0]
return self._languages[0][0]
@property
def chaos(self) -> float:
return self._mean_mess_ratio
@property
def coherence(self) -> float:
if not self._languages:
return 0.0
return self._languages[0][1]
@property
def percent_chaos(self) -> float:
return round(self.chaos * 100, ndigits=3)
@property
def percent_coherence(self) -> float:
return round(self.coherence * 100, ndigits=3)
@property
def raw(self) -> bytes | bytearray:
"""
Original untouched bytes.
"""
return self._payload
@property
def submatch(self) -> list[CharsetMatch]:
return self._leaves
@property
def has_submatch(self) -> bool:
return len(self._leaves) > 0
@property
def alphabets(self) -> list[str]:
if self._unicode_ranges is not None:
return self._unicode_ranges
# list detected ranges
detected_ranges: list[str | None] = [unicode_range(char) for char in str(self)]
# filter and sort
self._unicode_ranges = sorted(list({r for r in detected_ranges if r}))
return self._unicode_ranges
@property
def could_be_from_charset(self) -> list[str]:
"""
The complete list of encoding that output the exact SAME str result and therefore could be the originating
encoding.
This list does include the encoding available in property 'encoding'.
"""
return [self._encoding] + [m.encoding for m in self._leaves]
def output(self, encoding: str = "utf_8") -> bytes:
"""
Method to get re-encoded bytes payload using given target encoding. Default to UTF-8.
Any errors will be simply ignored by the encoder NOT replaced.
"""
if self._output_encoding is None or self._output_encoding != encoding:
self._output_encoding = encoding
decoded_string = str(self)
if (
self._preemptive_declaration is not None
and self._preemptive_declaration.lower()
not in ["utf-8", "utf8", "utf_8"]
):
patched_header = sub(
RE_POSSIBLE_ENCODING_INDICATION,
lambda m: m.string[m.span()[0] : m.span()[1]].replace(
m.groups()[0],
iana_name(self._output_encoding).replace("_", "-"), # type: ignore[arg-type]
),
decoded_string[:8192],
count=1,
)
decoded_string = patched_header + decoded_string[8192:]
self._output_payload = decoded_string.encode(encoding, "replace")
return self._output_payload # type: ignore
@property
def fingerprint(self) -> int:
"""
Retrieve a hash fingerprint of the decoded payload, used for deduplication.
"""
return hash(str(self))
class CharsetMatches:
"""
Container with every CharsetMatch items ordered by default from most probable to the less one.
Act like a list(iterable) but does not implements all related methods.
"""
def __init__(self, results: list[CharsetMatch] | None = None):
self._results: list[CharsetMatch] = sorted(results) if results else []
def __iter__(self) -> Iterator[CharsetMatch]:
yield from self._results
def __getitem__(self, item: int | str) -> CharsetMatch:
"""
Retrieve a single item either by its position or encoding name (alias may be used here).
Raise KeyError upon invalid index or encoding not present in results.
"""
if isinstance(item, int):
return self._results[item]
if isinstance(item, str):
item = iana_name(item, False)
for result in self._results:
if item in result.could_be_from_charset:
return result
raise KeyError
def __len__(self) -> int:
return len(self._results)
def __bool__(self) -> bool:
return len(self._results) > 0
def append(self, item: CharsetMatch) -> None:
"""
Insert a single match. Will be inserted accordingly to preserve sort.
Can be inserted as a submatch.
"""
if not isinstance(item, CharsetMatch):
raise ValueError(
"Cannot append instance '{}' to CharsetMatches".format(
str(item.__class__)
)
)
# We should disable the submatch factoring when the input file is too heavy (conserve RAM usage)
if len(item.raw) < TOO_BIG_SEQUENCE:
for match in self._results:
if match.fingerprint == item.fingerprint and match.chaos == item.chaos:
match.add_submatch(item)
return
self._results.append(item)
self._results = sorted(self._results)
def best(self) -> CharsetMatch | None:
"""
Simply return the first match. Strict equivalent to matches[0].
"""
if not self._results:
return None
return self._results[0]
def first(self) -> CharsetMatch | None:
"""
Redundant method, call the method best(). Kept for BC reasons.
"""
return self.best()
CoherenceMatch = Tuple[str, float]
CoherenceMatches = List[CoherenceMatch]
class CliDetectionResult:
def __init__(
self,
path: str,
encoding: str | None,
encoding_aliases: list[str],
alternative_encodings: list[str],
language: str,
alphabets: list[str],
has_sig_or_bom: bool,
chaos: float,
coherence: float,
unicode_path: str | None,
is_preferred: bool,
):
self.path: str = path
self.unicode_path: str | None = unicode_path
self.encoding: str | None = encoding
self.encoding_aliases: list[str] = encoding_aliases
self.alternative_encodings: list[str] = alternative_encodings
self.language: str = language
self.alphabets: list[str] = alphabets
self.has_sig_or_bom: bool = has_sig_or_bom
self.chaos: float = chaos
self.coherence: float = coherence
self.is_preferred: bool = is_preferred
@property
def __dict__(self) -> dict[str, Any]: # type: ignore
return {
"path": self.path,
"encoding": self.encoding,
"encoding_aliases": self.encoding_aliases,
"alternative_encodings": self.alternative_encodings,
"language": self.language,
"alphabets": self.alphabets,
"has_sig_or_bom": self.has_sig_or_bom,
"chaos": self.chaos,
"coherence": self.coherence,
"unicode_path": self.unicode_path,
"is_preferred": self.is_preferred,
}
def to_json(self) -> str:
return dumps(self.__dict__, ensure_ascii=True, indent=4)

0
.venv/lib/python3.12/site-packages/charset_normalizer/py.typed Normal file
View File

422
.venv/lib/python3.12/site-packages/charset_normalizer/utils.py Normal file
View File

@@ -0,0 +1,422 @@
from __future__ import annotations
import importlib
import logging
import unicodedata
from bisect import bisect_right
from codecs import IncrementalDecoder
from encodings.aliases import aliases
from functools import lru_cache
from re import findall
from typing import Generator
from _multibytecodec import ( # type: ignore[import-not-found,import]
MultibyteIncrementalDecoder,
)
from .constant import (
ENCODING_MARKS,
IANA_SUPPORTED_SIMILAR,
RE_POSSIBLE_ENCODING_INDICATION,
UNICODE_RANGES_COMBINED,
UNICODE_SECONDARY_RANGE_KEYWORD,
UTF8_MAXIMAL_ALLOCATION,
COMMON_CJK_CHARACTERS,
_LATIN,
_CJK,
_HANGUL,
_KATAKANA,
_HIRAGANA,
_THAI,
_ARABIC,
_ARABIC_ISOLATED_FORM,
_ACCENT_KEYWORDS,
_ACCENTUATED,
)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def _character_flags(character: str) -> int:
"""Compute all name-based classification flags with a single unicodedata.name() call."""
try:
desc: str = unicodedata.name(character)
except ValueError:
return 0
flags: int = 0
if "LATIN" in desc:
flags |= _LATIN
if "CJK" in desc:
flags |= _CJK
if "HANGUL" in desc:
flags |= _HANGUL
if "KATAKANA" in desc:
flags |= _KATAKANA
if "HIRAGANA" in desc:
flags |= _HIRAGANA
if "THAI" in desc:
flags |= _THAI
if "ARABIC" in desc:
flags |= _ARABIC
if "ISOLATED FORM" in desc:
flags |= _ARABIC_ISOLATED_FORM
for kw in _ACCENT_KEYWORDS:
if kw in desc:
flags |= _ACCENTUATED
break
return flags
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_accentuated(character: str) -> bool:
return bool(_character_flags(character) & _ACCENTUATED)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def remove_accent(character: str) -> str:
decomposed: str = unicodedata.decomposition(character)
if not decomposed:
return character
codes: list[str] = decomposed.split(" ")
return chr(int(codes[0], 16))
# Pre-built sorted lookup table for O(log n) binary search in unicode_range().
# Each entry is (range_start, range_end_exclusive, range_name).
_UNICODE_RANGES_SORTED: list[tuple[int, int, str]] = sorted(
(ord_range.start, ord_range.stop, name)
for name, ord_range in UNICODE_RANGES_COMBINED.items()
)
_UNICODE_RANGE_STARTS: list[int] = [e[0] for e in _UNICODE_RANGES_SORTED]
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def unicode_range(character: str) -> str | None:
"""
Retrieve the Unicode range official name from a single character.
"""
character_ord: int = ord(character)
# Binary search: find the rightmost range whose start <= character_ord
idx = bisect_right(_UNICODE_RANGE_STARTS, character_ord) - 1
if idx >= 0:
start, stop, name = _UNICODE_RANGES_SORTED[idx]
if character_ord < stop:
return name
return None
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_latin(character: str) -> bool:
return bool(_character_flags(character) & _LATIN)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_punctuation(character: str) -> bool:
character_category: str = unicodedata.category(character)
if "P" in character_category:
return True
character_range: str | None = unicode_range(character)
if character_range is None:
return False
return "Punctuation" in character_range
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_symbol(character: str) -> bool:
character_category: str = unicodedata.category(character)
if "S" in character_category or "N" in character_category:
return True
character_range: str | None = unicode_range(character)
if character_range is None:
return False
return "Forms" in character_range and character_category != "Lo"
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_emoticon(character: str) -> bool:
character_range: str | None = unicode_range(character)
if character_range is None:
return False
return "Emoticons" in character_range or "Pictographs" in character_range
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_separator(character: str) -> bool:
if character.isspace() or character in {"｜", "+", "<", ">"}:
return True
character_category: str = unicodedata.category(character)
return "Z" in character_category or character_category in {"Po", "Pd", "Pc"}
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_case_variable(character: str) -> bool:
return character.islower() != character.isupper()
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_cjk(character: str) -> bool:
return bool(_character_flags(character) & _CJK)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_hiragana(character: str) -> bool:
return bool(_character_flags(character) & _HIRAGANA)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_katakana(character: str) -> bool:
return bool(_character_flags(character) & _KATAKANA)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_hangul(character: str) -> bool:
return bool(_character_flags(character) & _HANGUL)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_thai(character: str) -> bool:
return bool(_character_flags(character) & _THAI)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_arabic(character: str) -> bool:
return bool(_character_flags(character) & _ARABIC)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_arabic_isolated_form(character: str) -> bool:
return bool(_character_flags(character) & _ARABIC_ISOLATED_FORM)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_cjk_uncommon(character: str) -> bool:
return character not in COMMON_CJK_CHARACTERS
@lru_cache(maxsize=len(UNICODE_RANGES_COMBINED))
def is_unicode_range_secondary(range_name: str) -> bool:
return any(keyword in range_name for keyword in UNICODE_SECONDARY_RANGE_KEYWORD)
@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
def is_unprintable(character: str) -> bool:
return (
character.isspace() is False # includes \n \t \r \v
and character.isprintable() is False
and character != "\x1a" # Why? Its the ASCII substitute character.
and character != "\ufeff" # bug discovered in Python,
# Zero Width No-Break Space located in Arabic Presentation Forms-B, Unicode 1.1 not acknowledged as space.
)
def any_specified_encoding(
sequence: bytes | bytearray, search_zone: int = 8192
) -> str | None:
"""
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
"""
if not isinstance(sequence, (bytes, bytearray)):
raise TypeError
seq_len: int = len(sequence)
results: list[str] = findall(
RE_POSSIBLE_ENCODING_INDICATION,
sequence[: min(seq_len, search_zone)].decode("ascii", errors="ignore"),
)
if len(results) == 0:
return None
for specified_encoding in results:
specified_encoding = specified_encoding.lower().replace("-", "_")
encoding_alias: str
encoding_iana: str
for encoding_alias, encoding_iana in aliases.items():
if encoding_alias == specified_encoding:
return encoding_iana
if encoding_iana == specified_encoding:
return encoding_iana
return None
@lru_cache(maxsize=128)
def is_multi_byte_encoding(name: str) -> bool:
"""
Verify is a specific encoding is a multi byte one based on it IANA name
"""
return name in {
"utf_8",
"utf_8_sig",
"utf_16",
"utf_16_be",
"utf_16_le",
"utf_32",
"utf_32_le",
"utf_32_be",
"utf_7",
} or issubclass(
importlib.import_module(f"encodings.{name}").IncrementalDecoder,
MultibyteIncrementalDecoder,
)
def identify_sig_or_bom(sequence: bytes | bytearray) -> tuple[str | None, bytes]:
"""
Identify and extract SIG/BOM in given sequence.
"""
for iana_encoding in ENCODING_MARKS:
marks: bytes | list[bytes] = ENCODING_MARKS[iana_encoding]
if isinstance(marks, bytes):
marks = [marks]
for mark in marks:
if sequence.startswith(mark):
return iana_encoding, mark
return None, b""
def should_strip_sig_or_bom(iana_encoding: str) -> bool:
return iana_encoding not in {"utf_16", "utf_32"}
def iana_name(cp_name: str, strict: bool = True) -> str:
"""Returns the Python normalized encoding name (Not the IANA official name)."""
cp_name = cp_name.lower().replace("-", "_")
encoding_alias: str
encoding_iana: str
for encoding_alias, encoding_iana in aliases.items():
if cp_name in [encoding_alias, encoding_iana]:
return encoding_iana
if strict:
raise ValueError(f"Unable to retrieve IANA for '{cp_name}'")
return cp_name
def cp_similarity(iana_name_a: str, iana_name_b: str) -> float:
if is_multi_byte_encoding(iana_name_a) or is_multi_byte_encoding(iana_name_b):
return 0.0
decoder_a = importlib.import_module(f"encodings.{iana_name_a}").IncrementalDecoder
decoder_b = importlib.import_module(f"encodings.{iana_name_b}").IncrementalDecoder
id_a: IncrementalDecoder = decoder_a(errors="ignore")
id_b: IncrementalDecoder = decoder_b(errors="ignore")
character_match_count: int = 0
for i in range(256):
to_be_decoded: bytes = bytes([i])
if id_a.decode(to_be_decoded) == id_b.decode(to_be_decoded):
character_match_count += 1
return character_match_count / 256
def is_cp_similar(iana_name_a: str, iana_name_b: str) -> bool:
"""
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
"""
return (
iana_name_a in IANA_SUPPORTED_SIMILAR
and iana_name_b in IANA_SUPPORTED_SIMILAR[iana_name_a]
)
def set_logging_handler(
name: str = "charset_normalizer",
level: int = logging.INFO,
format_string: str = "%(asctime)s | %(levelname)s | %(message)s",
) -> None:
logger = logging.getLogger(name)
logger.setLevel(level)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(format_string))
logger.addHandler(handler)
def cut_sequence_chunks(
sequences: bytes | bytearray,
encoding_iana: str,
offsets: range,
chunk_size: int,
bom_or_sig_available: bool,
strip_sig_or_bom: bool,
sig_payload: bytes,
is_multi_byte_decoder: bool,
decoded_payload: str | None = None,
) -> Generator[str, None, None]:
if decoded_payload and is_multi_byte_decoder is False:
for i in offsets:
chunk = decoded_payload[i : i + chunk_size]
if not chunk:
break
yield chunk
else:
for i in offsets:
chunk_end = i + chunk_size
if chunk_end > len(sequences) + 8:
continue
cut_sequence = sequences[i : i + chunk_size]
if bom_or_sig_available and strip_sig_or_bom is False:
cut_sequence = sig_payload + cut_sequence
chunk = cut_sequence.decode(
encoding_iana,
errors="ignore" if is_multi_byte_decoder else "strict",
)
# multi-byte bad cutting detector and adjustment
# not the cleanest way to perform that fix but clever enough for now.
if is_multi_byte_decoder and i > 0:
chunk_partial_size_chk: int = min(chunk_size, 16)
if (
decoded_payload
and chunk[:chunk_partial_size_chk] not in decoded_payload
):
for j in range(i, i - 4, -1):
cut_sequence = sequences[j:chunk_end]
if bom_or_sig_available and strip_sig_or_bom is False:
cut_sequence = sig_payload + cut_sequence
chunk = cut_sequence.decode(encoding_iana, errors="ignore")
if chunk[:chunk_partial_size_chk] in decoded_payload:
break
yield chunk

8
.venv/lib/python3.12/site-packages/charset_normalizer/version.py Normal file
View File

@@ -0,0 +1,8 @@
"""
Expose version
"""
from __future__ import annotations
__version__ = "3.4.7"
VERSION = __version__.split(".")

51
.venv/lib/python3.12/site-packages/dotenv/__init__.py Normal file
View File

@@ -0,0 +1,51 @@
from typing import Any, Optional
from .main import dotenv_values, find_dotenv, get_key, load_dotenv, set_key, unset_key
def load_ipython_extension(ipython: Any) -> None:
from .ipython import load_ipython_extension
load_ipython_extension(ipython)
def get_cli_string(
path: Optional[str] = None,
action: Optional[str] = None,
key: Optional[str] = None,
value: Optional[str] = None,
quote: Optional[str] = None,
):
"""Returns a string suitable for running as a shell script.
Useful for converting a arguments passed to a fabric task
to be passed to a `local` or `run` command.
"""
command = ["dotenv"]
if quote:
command.append(f"-q {quote}")
if path:
command.append(f"-f {path}")
if action:
command.append(action)
if key:
command.append(key)
if value:
if " " in value:
command.append(f'"{value}"')
else:
command.append(value)
return " ".join(command).strip()
__all__ = [
"get_cli_string",
"load_dotenv",
"dotenv_values",
"get_key",
"set_key",
"unset_key",
"find_dotenv",
"load_ipython_extension",
]

6
.venv/lib/python3.12/site-packages/dotenv/__main__.py Normal file
View File

@@ -0,0 +1,6 @@
"""Entry point for cli, enables execution with `python -m dotenv`"""
from .cli import cli
if __name__ == "__main__":
cli()

BIN
.venv/lib/python3.12/site-packages/dotenv/__pycache__/__init__.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/dotenv/__pycache__/__main__.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/dotenv/__pycache__/cli.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/dotenv/__pycache__/ipython.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/dotenv/__pycache__/main.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/dotenv/__pycache__/parser.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/dotenv/__pycache__/variables.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/dotenv/__pycache__/version.cpython-312.pyc Normal file
View File

Binary file not shown.

236
.venv/lib/python3.12/site-packages/dotenv/cli.py Normal file
View File

@@ -0,0 +1,236 @@
import json
import os
import shlex
import sys
from contextlib import contextmanager
from typing import IO, Any, Dict, Iterator, List, Optional
if sys.platform == "win32":
from subprocess import Popen
try:
import click
except ImportError:
sys.stderr.write(
"It seems python-dotenv is not installed with cli option. \n"
'Run pip install "python-dotenv[cli]" to fix this.'
)
sys.exit(1)
from .main import dotenv_values, set_key, unset_key
from .version import __version__
def enumerate_env() -> Optional[str]:
"""
Return a path for the ${pwd}/.env file.
If pwd does not exist, return None.
"""
try:
cwd = os.getcwd()
except FileNotFoundError:
return None
path = os.path.join(cwd, ".env")
return path
@click.group()
@click.option(
"-f",
"--file",
default=enumerate_env(),
type=click.Path(file_okay=True),
help="Location of the .env file, defaults to .env file in current working directory.",
)
@click.option(
"-q",
"--quote",
default="always",
type=click.Choice(["always", "never", "auto"]),
help="Whether to quote or not the variable values. Default mode is always. This does not affect parsing.",
)
@click.option(
"-e",
"--export",
default=False,
type=click.BOOL,
help="Whether to write the dot file as an executable bash script.",
)
@click.version_option(version=__version__)
@click.pass_context
def cli(ctx: click.Context, file: Any, quote: Any, export: Any) -> None:
"""This script is used to set, get or unset values from a .env file."""
ctx.obj = {"QUOTE": quote, "EXPORT": export, "FILE": file}
@contextmanager
def stream_file(path: os.PathLike) -> Iterator[IO[str]]:
"""
Open a file and yield the corresponding (decoded) stream.
Exits with error code 2 if the file cannot be opened.
"""
try:
with open(path) as stream:
yield stream
except OSError as exc:
print(f"Error opening env file: {exc}", file=sys.stderr)
sys.exit(2)
@cli.command(name="list")
@click.pass_context
@click.option(
"--format",
"output_format",
default="simple",
type=click.Choice(["simple", "json", "shell", "export"]),
help="The format in which to display the list. Default format is simple, "
"which displays name=value without quotes.",
)
def list_values(ctx: click.Context, output_format: str) -> None:
"""Display all the stored key/value."""
file = ctx.obj["FILE"]
with stream_file(file) as stream:
values = dotenv_values(stream=stream)
if output_format == "json":
click.echo(json.dumps(values, indent=2, sort_keys=True))
else:
prefix = "export " if output_format == "export" else ""
for k in sorted(values):
v = values[k]
if v is not None:
if output_format in ("export", "shell"):
v = shlex.quote(v)
click.echo(f"{prefix}{k}={v}")
@cli.command(name="set")
@click.pass_context
@click.argument("key", required=True)
@click.argument("value", required=True)
def set_value(ctx: click.Context, key: Any, value: Any) -> None:
"""
Store the given key/value.
This doesn't follow symlinks, to avoid accidentally modifying a file at a
potentially untrusted path.
"""
file = ctx.obj["FILE"]
quote = ctx.obj["QUOTE"]
export = ctx.obj["EXPORT"]
success, key, value = set_key(file, key, value, quote, export)
if success:
click.echo(f"{key}={value}")
else:
sys.exit(1)
@cli.command()
@click.pass_context
@click.argument("key", required=True)
def get(ctx: click.Context, key: Any) -> None:
"""Retrieve the value for the given key."""
file = ctx.obj["FILE"]
with stream_file(file) as stream:
values = dotenv_values(stream=stream)
stored_value = values.get(key)
if stored_value:
click.echo(stored_value)
else:
sys.exit(1)
@cli.command()
@click.pass_context
@click.argument("key", required=True)
def unset(ctx: click.Context, key: Any) -> None:
"""
Removes the given key.
This doesn't follow symlinks, to avoid accidentally modifying a file at a
potentially untrusted path.
"""
file = ctx.obj["FILE"]
quote = ctx.obj["QUOTE"]
success, key = unset_key(file, key, quote)
if success:
click.echo(f"Successfully removed {key}")
else:
sys.exit(1)
@cli.command(
context_settings={
"allow_extra_args": True,
"allow_interspersed_args": False,
"ignore_unknown_options": True,
}
)
@click.pass_context
@click.option(
"--override/--no-override",
default=True,
help="Override variables from the environment file with those from the .env file.",
)
@click.argument("commandline", nargs=-1, type=click.UNPROCESSED)
def run(ctx: click.Context, override: bool, commandline: tuple[str, ...]) -> None:
"""Run command with environment variables present."""
file = ctx.obj["FILE"]
if not os.path.isfile(file):
raise click.BadParameter(
f"Invalid value for '-f' \"{file}\" does not exist.", ctx=ctx
)
dotenv_as_dict = {
k: v
for (k, v) in dotenv_values(file).items()
if v is not None and (override or k not in os.environ)
}
if not commandline:
click.echo("No command given.")
sys.exit(1)
run_command([*commandline, *ctx.args], dotenv_as_dict)
def run_command(command: List[str], env: Dict[str, str]) -> None:
"""Replace the current process with the specified command.
Replaces the current process with the specified command and the variables from `env`
added in the current environment variables.
Parameters
----------
command: List[str]
The command and it's parameters
env: Dict
The additional environment variables
Returns
-------
None
This function does not return any value. It replaces the current process with the new one.
"""
# copy the current environment variables and add the vales from
# `env`
cmd_env = os.environ.copy()
cmd_env.update(env)
if sys.platform == "win32":
# execvpe on Windows returns control immediately
# rather than once the command has finished.
p = Popen(command, universal_newlines=True, bufsize=0, shell=False, env=cmd_env)
_, _ = p.communicate()
sys.exit(p.returncode)
else:
os.execvpe(command[0], args=command, env=cmd_env)

50
.venv/lib/python3.12/site-packages/dotenv/ipython.py Normal file
View File

@@ -0,0 +1,50 @@
from IPython.core.magic import Magics, line_magic, magics_class # type: ignore
from IPython.core.magic_arguments import (
argument,
magic_arguments,
parse_argstring,
) # type: ignore
from .main import find_dotenv, load_dotenv
@magics_class
class IPythonDotEnv(Magics):
@magic_arguments()
@argument(
"-o",
"--override",
action="store_true",
help="Indicate to override existing variables",
)
@argument(
"-v",
"--verbose",
action="store_true",
help="Indicate function calls to be verbose",
)
@argument(
"dotenv_path",
nargs="?",
type=str,
default=".env",
help="Search in increasingly higher folders for the `dotenv_path`",
)
@line_magic
def dotenv(self, line):
args = parse_argstring(self.dotenv, line)
# Locate the .env file
dotenv_path = args.dotenv_path
try:
dotenv_path = find_dotenv(dotenv_path, True, True)
except IOError:
print("cannot find .env file")
return
# Load the .env file
load_dotenv(dotenv_path, verbose=args.verbose, override=args.override)
def load_ipython_extension(ipython):
"""Register the %dotenv magic."""
ipython.register_magics(IPythonDotEnv)

480
.venv/lib/python3.12/site-packages/dotenv/main.py Normal file
View File

@@ -0,0 +1,480 @@
import io
import logging
import os
import pathlib
import stat
import sys
import tempfile
from collections import OrderedDict
from contextlib import contextmanager
from typing import IO, Dict, Iterable, Iterator, Mapping, Optional, Tuple, Union
from .parser import Binding, parse_stream
from .variables import parse_variables
# A type alias for a string path to be used for the paths in this file.
# These paths may flow to `open()` and `os.replace()`.
StrPath = Union[str, "os.PathLike[str]"]
logger = logging.getLogger(__name__)
def _load_dotenv_disabled() -> bool:
"""
Determine if dotenv loading has been disabled.
"""
if "PYTHON_DOTENV_DISABLED" not in os.environ:
return False
value = os.environ["PYTHON_DOTENV_DISABLED"].casefold()
return value in {"1", "true", "t", "yes", "y"}
def with_warn_for_invalid_lines(mappings: Iterator[Binding]) -> Iterator[Binding]:
for mapping in mappings:
if mapping.error:
logger.warning(
"python-dotenv could not parse statement starting at line %s",
mapping.original.line,
)
yield mapping
class DotEnv:
def __init__(
self,
dotenv_path: Optional[StrPath],
stream: Optional[IO[str]] = None,
verbose: bool = False,
encoding: Optional[str] = None,
interpolate: bool = True,
override: bool = True,
) -> None:
self.dotenv_path: Optional[StrPath] = dotenv_path
self.stream: Optional[IO[str]] = stream
self._dict: Optional[Dict[str, Optional[str]]] = None
self.verbose: bool = verbose
self.encoding: Optional[str] = encoding
self.interpolate: bool = interpolate
self.override: bool = override
@contextmanager
def _get_stream(self) -> Iterator[IO[str]]:
if self.dotenv_path and _is_file_or_fifo(self.dotenv_path):
with open(self.dotenv_path, encoding=self.encoding) as stream:
yield stream
elif self.stream is not None:
yield self.stream
else:
if self.verbose:
logger.info(
"python-dotenv could not find configuration file %s.",
self.dotenv_path or ".env",
)
yield io.StringIO("")
def dict(self) -> Dict[str, Optional[str]]:
"""Return dotenv as dict"""
if self._dict:
return self._dict
raw_values = self.parse()
if self.interpolate:
self._dict = OrderedDict(
resolve_variables(raw_values, override=self.override)
)
else:
self._dict = OrderedDict(raw_values)
return self._dict
def parse(self) -> Iterator[Tuple[str, Optional[str]]]:
with self._get_stream() as stream:
for mapping in with_warn_for_invalid_lines(parse_stream(stream)):
if mapping.key is not None:
yield mapping.key, mapping.value
def set_as_environment_variables(self) -> bool:
"""
Load the current dotenv as system environment variable.
"""
if not self.dict():
return False
for k, v in self.dict().items():
if k in os.environ and not self.override:
continue
if v is not None:
os.environ[k] = v
return True
def get(self, key: str) -> Optional[str]:
""" """
data = self.dict()
if key in data:
return data[key]
if self.verbose:
logger.warning("Key %s not found in %s.", key, self.dotenv_path)
return None
def get_key(
dotenv_path: StrPath,
key_to_get: str,
encoding: Optional[str] = "utf-8",
) -> Optional[str]:
"""
Get the value of a given key from the given .env.
Returns `None` if the key isn't found or doesn't have a value.
"""
return DotEnv(dotenv_path, verbose=True, encoding=encoding).get(key_to_get)
@contextmanager
def rewrite(
path: StrPath,
encoding: Optional[str],
follow_symlinks: bool = False,
) -> Iterator[Tuple[IO[str], IO[str]]]:
if follow_symlinks:
path = os.path.realpath(path)
try:
source: IO[str] = open(path, encoding=encoding)
try:
path_stat = os.lstat(path)
original_mode: Optional[int] = (
stat.S_IMODE(path_stat.st_mode)
if stat.S_ISREG(path_stat.st_mode)
else None
)
except BaseException:
source.close()
raise
except FileNotFoundError:
source = io.StringIO("")
original_mode = None
with tempfile.NamedTemporaryFile(
mode="w",
encoding=encoding,
delete=False,
prefix=".tmp_",
dir=os.path.dirname(os.path.abspath(path)),
) as dest:
dest_path = pathlib.Path(dest.name)
error = None
try:
with source:
yield (source, dest)
except BaseException as err:
error = err
if error is None:
try:
if original_mode is not None:
os.chmod(dest_path, original_mode)
os.replace(dest_path, path)
except BaseException:
dest_path.unlink(missing_ok=True)
raise
else:
dest_path.unlink(missing_ok=True)
raise error from None
def set_key(
dotenv_path: StrPath,
key_to_set: str,
value_to_set: str,
quote_mode: str = "always",
export: bool = False,
encoding: Optional[str] = "utf-8",
follow_symlinks: bool = False,
) -> Tuple[Optional[bool], str, str]:
"""
Adds or Updates a key/value to the given .env
The target .env file is created if it doesn't exist.
This function doesn't follow symlinks by default, to avoid accidentally
modifying a file at a potentially untrusted path. If you don't need this
protection and need symlinks to be followed, use `follow_symlinks`.
"""
if quote_mode not in ("always", "auto", "never"):
raise ValueError(f"Unknown quote_mode: {quote_mode}")
quote = quote_mode == "always" or (
quote_mode == "auto" and not value_to_set.isalnum()
)
if quote:
value_out = "'{}'".format(value_to_set.replace("'", "\\'"))
else:
value_out = value_to_set
if export:
line_out = f"export {key_to_set}={value_out}\n"
else:
line_out = f"{key_to_set}={value_out}\n"
with rewrite(dotenv_path, encoding=encoding, follow_symlinks=follow_symlinks) as (
source,
dest,
):
replaced = False
missing_newline = False
for mapping in with_warn_for_invalid_lines(parse_stream(source)):
if mapping.key == key_to_set:
dest.write(line_out)
replaced = True
else:
dest.write(mapping.original.string)
missing_newline = not mapping.original.string.endswith("\n")
if not replaced:
if missing_newline:
dest.write("\n")
dest.write(line_out)
return True, key_to_set, value_to_set
def unset_key(
dotenv_path: StrPath,
key_to_unset: str,
quote_mode: str = "always",
encoding: Optional[str] = "utf-8",
follow_symlinks: bool = False,
) -> Tuple[Optional[bool], str]:
"""
Removes a given key from the given `.env` file.
If the .env path given doesn't exist, fails.
If the given key doesn't exist in the .env, fails.
This function doesn't follow symlinks by default, to avoid accidentally
modifying a file at a potentially untrusted path. If you don't need this
protection and need symlinks to be followed, use `follow_symlinks`.
"""
if not os.path.exists(dotenv_path):
logger.warning("Can't delete from %s - it doesn't exist.", dotenv_path)
return None, key_to_unset
removed = False
with rewrite(dotenv_path, encoding=encoding, follow_symlinks=follow_symlinks) as (
source,
dest,
):
for mapping in with_warn_for_invalid_lines(parse_stream(source)):
if mapping.key == key_to_unset:
removed = True
else:
dest.write(mapping.original.string)
if not removed:
logger.warning(
"Key %s not removed from %s - key doesn't exist.", key_to_unset, dotenv_path
)
return None, key_to_unset
return removed, key_to_unset
def resolve_variables(
values: Iterable[Tuple[str, Optional[str]]],
override: bool,
) -> Mapping[str, Optional[str]]:
new_values: Dict[str, Optional[str]] = {}
for name, value in values:
if value is None:
result = None
else:
atoms = parse_variables(value)
env: Dict[str, Optional[str]] = {}
if override:
env.update(os.environ) # type: ignore
env.update(new_values)
else:
env.update(new_values)
env.update(os.environ) # type: ignore
result = "".join(atom.resolve(env) for atom in atoms)
new_values[name] = result
return new_values
def _walk_to_root(path: str) -> Iterator[str]:
"""
Yield directories starting from the given directory up to the root
"""
if not os.path.exists(path):
raise IOError("Starting path not found")
if os.path.isfile(path):
path = os.path.dirname(path)
last_dir = None
current_dir = os.path.abspath(path)
while last_dir != current_dir:
yield current_dir
parent_dir = os.path.abspath(os.path.join(current_dir, os.path.pardir))
last_dir, current_dir = current_dir, parent_dir
def find_dotenv(
filename: str = ".env",
raise_error_if_not_found: bool = False,
usecwd: bool = False,
) -> str:
"""
Search in increasingly higher folders for the given file
Returns path to the file if found, or an empty string otherwise
"""
def _is_interactive():
"""Decide whether this is running in a REPL or IPython notebook"""
if hasattr(sys, "ps1") or hasattr(sys, "ps2"):
return True
try:
main = __import__("__main__", None, None, fromlist=["__file__"])
except ModuleNotFoundError:
return False
return not hasattr(main, "__file__")
def _is_debugger():
return sys.gettrace() is not None
if usecwd or _is_interactive() or _is_debugger() or getattr(sys, "frozen", False):
# Should work without __file__, e.g. in REPL or IPython notebook.
path = os.getcwd()
else:
# will work for .py files
frame = sys._getframe()
current_file = __file__
while frame.f_code.co_filename == current_file or not os.path.exists(
frame.f_code.co_filename
):
assert frame.f_back is not None
frame = frame.f_back
frame_filename = frame.f_code.co_filename
path = os.path.dirname(os.path.abspath(frame_filename))
for dirname in _walk_to_root(path):
check_path = os.path.join(dirname, filename)
if _is_file_or_fifo(check_path):
return check_path
if raise_error_if_not_found:
raise IOError("File not found")
return ""
def load_dotenv(
dotenv_path: Optional[StrPath] = None,
stream: Optional[IO[str]] = None,
verbose: bool = False,
override: bool = False,
interpolate: bool = True,
encoding: Optional[str] = "utf-8",
) -> bool:
"""Parse a .env file and then load all the variables found as environment variables.
Parameters:
dotenv_path: Absolute or relative path to .env file.
stream: Text stream (such as `io.StringIO`) with .env content, used if
`dotenv_path` is `None`.
verbose: Whether to output a warning the .env file is missing.
override: Whether to override the system environment variables with the variables
from the `.env` file.
encoding: Encoding to be used to read the file.
Returns:
Bool: True if at least one environment variable is set else False
If both `dotenv_path` and `stream` are `None`, `find_dotenv()` is used to find the
.env file with it's default parameters. If you need to change the default parameters
of `find_dotenv()`, you can explicitly call `find_dotenv()` and pass the result
to this function as `dotenv_path`.
If the environment variable `PYTHON_DOTENV_DISABLED` is set to a truthy value,
.env loading is disabled.
"""
if _load_dotenv_disabled():
logger.debug(
"python-dotenv: .env loading disabled by PYTHON_DOTENV_DISABLED environment variable"
)
return False
if dotenv_path is None and stream is None:
dotenv_path = find_dotenv()
dotenv = DotEnv(
dotenv_path=dotenv_path,
stream=stream,
verbose=verbose,
interpolate=interpolate,
override=override,
encoding=encoding,
)
return dotenv.set_as_environment_variables()
def dotenv_values(
dotenv_path: Optional[StrPath] = None,
stream: Optional[IO[str]] = None,
verbose: bool = False,
interpolate: bool = True,
encoding: Optional[str] = "utf-8",
) -> Dict[str, Optional[str]]:
"""
Parse a .env file and return its content as a dict.
The returned dict will have `None` values for keys without values in the .env file.
For example, `foo=bar` results in `{"foo": "bar"}` whereas `foo` alone results in
`{"foo": None}`
Parameters:
dotenv_path: Absolute or relative path to the .env file.
stream: `StringIO` object with .env content, used if `dotenv_path` is `None`.
verbose: Whether to output a warning if the .env file is missing.
encoding: Encoding to be used to read the file.
If both `dotenv_path` and `stream` are `None`, `find_dotenv()` is used to find the
.env file.
"""
if dotenv_path is None and stream is None:
dotenv_path = find_dotenv()
return DotEnv(
dotenv_path=dotenv_path,
stream=stream,
verbose=verbose,
interpolate=interpolate,
override=True,
encoding=encoding,
).dict()
def _is_file_or_fifo(path: StrPath) -> bool:
"""
Return True if `path` exists and is either a regular file or a FIFO.
"""
if os.path.isfile(path):
return True
try:
st = os.stat(path)
except (FileNotFoundError, OSError):
return False
return stat.S_ISFIFO(st.st_mode)

182
.venv/lib/python3.12/site-packages/dotenv/parser.py Normal file
View File

@@ -0,0 +1,182 @@
import codecs
import re
from typing import (
IO,
Iterator,
Match,
NamedTuple,
Optional,
Pattern,
Sequence,
)
def make_regex(string: str, extra_flags: int = 0) -> Pattern[str]:
return re.compile(string, re.UNICODE | extra_flags)
_newline = make_regex(r"(\r\n|\n|\r)")
_multiline_whitespace = make_regex(r"\s*", extra_flags=re.MULTILINE)
_whitespace = make_regex(r"[^\S\r\n]*")
_export = make_regex(r"(?:export[^\S\r\n]+)?")
_single_quoted_key = make_regex(r"'([^']+)'")
_unquoted_key = make_regex(r"([^=\#\s]+)")
_equal_sign = make_regex(r"(=[^\S\r\n]*)")
_single_quoted_value = make_regex(r"'((?:\\'|[^'])*)'")
_double_quoted_value = make_regex(r'"((?:\\"|[^"])*)"')
_unquoted_value = make_regex(r"([^\r\n]*)")
_comment = make_regex(r"(?:[^\S\r\n]*#[^\r\n]*)?")
_end_of_line = make_regex(r"[^\S\r\n]*(?:\r\n|\n|\r|$)")
_rest_of_line = make_regex(r"[^\r\n]*(?:\r|\n|\r\n)?")
_double_quote_escapes = make_regex(r"\\[\\'\"abfnrtv]")
_single_quote_escapes = make_regex(r"\\[\\']")
class Original(NamedTuple):
string: str
line: int
class Binding(NamedTuple):
key: Optional[str]
value: Optional[str]
original: Original
error: bool
class Position:
def __init__(self, chars: int, line: int) -> None:
self.chars = chars
self.line = line
@classmethod
def start(cls) -> "Position":
return cls(chars=0, line=1)
def set(self, other: "Position") -> None:
self.chars = other.chars
self.line = other.line
def advance(self, string: str) -> None:
self.chars += len(string)
self.line += len(re.findall(_newline, string))
class Error(Exception):
pass
class Reader:
def __init__(self, stream: IO[str]) -> None:
self.string = stream.read()
self.position = Position.start()
self.mark = Position.start()
def has_next(self) -> bool:
return self.position.chars < len(self.string)
def set_mark(self) -> None:
self.mark.set(self.position)
def get_marked(self) -> Original:
return Original(
string=self.string[self.mark.chars : self.position.chars],
line=self.mark.line,
)
def peek(self, count: int) -> str:
return self.string[self.position.chars : self.position.chars + count]
def read(self, count: int) -> str:
result = self.string[self.position.chars : self.position.chars + count]
if len(result) < count:
raise Error("read: End of string")
self.position.advance(result)
return result
def read_regex(self, regex: Pattern[str]) -> Sequence[str]:
match = regex.match(self.string, self.position.chars)
if match is None:
raise Error("read_regex: Pattern not found")
self.position.advance(self.string[match.start() : match.end()])
return match.groups()
def decode_escapes(regex: Pattern[str], string: str) -> str:
def decode_match(match: Match[str]) -> str:
return codecs.decode(match.group(0), "unicode-escape") # type: ignore
return regex.sub(decode_match, string)
def parse_key(reader: Reader) -> Optional[str]:
char = reader.peek(1)
if char == "#":
return None
elif char == "'":
(key,) = reader.read_regex(_single_quoted_key)
else:
(key,) = reader.read_regex(_unquoted_key)
return key
def parse_unquoted_value(reader: Reader) -> str:
(part,) = reader.read_regex(_unquoted_value)
return re.sub(r"\s+#.*", "", part).rstrip()
def parse_value(reader: Reader) -> str:
char = reader.peek(1)
if char == "'":
(value,) = reader.read_regex(_single_quoted_value)
return decode_escapes(_single_quote_escapes, value)
elif char == '"':
(value,) = reader.read_regex(_double_quoted_value)
return decode_escapes(_double_quote_escapes, value)
elif char in ("", "\n", "\r"):
return ""
else:
return parse_unquoted_value(reader)
def parse_binding(reader: Reader) -> Binding:
reader.set_mark()
try:
reader.read_regex(_multiline_whitespace)
if not reader.has_next():
return Binding(
key=None,
value=None,
original=reader.get_marked(),
error=False,
)
reader.read_regex(_export)
key = parse_key(reader)
reader.read_regex(_whitespace)
if reader.peek(1) == "=":
reader.read_regex(_equal_sign)
value: Optional[str] = parse_value(reader)
else:
value = None
reader.read_regex(_comment)
reader.read_regex(_end_of_line)
return Binding(
key=key,
value=value,
original=reader.get_marked(),
error=False,
)
except Error:
reader.read_regex(_rest_of_line)
return Binding(
key=None,
value=None,
original=reader.get_marked(),
error=True,
)
def parse_stream(stream: IO[str]) -> Iterator[Binding]:
reader = Reader(stream)
while reader.has_next():
yield parse_binding(reader)

1
.venv/lib/python3.12/site-packages/dotenv/py.typed Normal file
View File

@@ -0,0 +1 @@
# Marker file for PEP 561

86
.venv/lib/python3.12/site-packages/dotenv/variables.py Normal file
View File

@@ -0,0 +1,86 @@
import re
from abc import ABCMeta, abstractmethod
from typing import Iterator, Mapping, Optional, Pattern
_posix_variable: Pattern[str] = re.compile(
r"""
\$\{
(?P<name>[^\}:]*)
(?::-
(?P<default>[^\}]*)
)?
\}
""",
re.VERBOSE,
)
class Atom(metaclass=ABCMeta):
def __ne__(self, other: object) -> bool:
result = self.__eq__(other)
if result is NotImplemented:
return NotImplemented
return not result
@abstractmethod
def resolve(self, env: Mapping[str, Optional[str]]) -> str: ...
class Literal(Atom):
def __init__(self, value: str) -> None:
self.value = value
def __repr__(self) -> str:
return f"Literal(value={self.value})"
def __eq__(self, other: object) -> bool:
if not isinstance(other, self.__class__):
return NotImplemented
return self.value == other.value
def __hash__(self) -> int:
return hash((self.__class__, self.value))
def resolve(self, env: Mapping[str, Optional[str]]) -> str:
return self.value
class Variable(Atom):
def __init__(self, name: str, default: Optional[str]) -> None:
self.name = name
self.default = default
def __repr__(self) -> str:
return f"Variable(name={self.name}, default={self.default})"
def __eq__(self, other: object) -> bool:
if not isinstance(other, self.__class__):
return NotImplemented
return (self.name, self.default) == (other.name, other.default)
def __hash__(self) -> int:
return hash((self.__class__, self.name, self.default))
def resolve(self, env: Mapping[str, Optional[str]]) -> str:
default = self.default if self.default is not None else ""
result = env.get(self.name, default)
return result if result is not None else ""
def parse_variables(value: str) -> Iterator[Atom]:
cursor = 0
for match in _posix_variable.finditer(value):
(start, end) = match.span()
name = match["name"]
default = match["default"]
if start > cursor:
yield Literal(value=value[cursor:start])
yield Variable(name=name, default=default)
cursor = end
length = len(value)
if cursor < length:
yield Literal(value=value[cursor:length])

1
.venv/lib/python3.12/site-packages/dotenv/version.py Normal file
View File

@@ -0,0 +1 @@
__version__ = "1.2.2"

1
.venv/lib/python3.12/site-packages/idna-3.18.dist-info/INSTALLER Normal file
View File

@@ -0,0 +1 @@
pip

155
.venv/lib/python3.12/site-packages/idna-3.18.dist-info/METADATA Normal file
View File

@@ -0,0 +1,155 @@
Metadata-Version: 2.4
Name: idna
Version: 3.18
Summary: Internationalized Domain Names in Applications (IDNA)
Author-email: Kim Davies <kim+pypi@gumleaf.org>
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-Expression: BSD-3-Clause
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Internet :: Name Service (DNS)
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
License-File: LICENSE.md
Requires-Dist: ruff >= 0.6.2 ; extra == "all"
Requires-Dist: mypy >= 1.11.2 ; extra == "all"
Requires-Dist: pytest >= 8.3.2 ; extra == "all"
Project-URL: Changelog, https://github.com/kjd/idna/blob/master/HISTORY.md
Project-URL: Issue tracker, https://github.com/kjd/idna/issues
Project-URL: Source, https://github.com/kjd/idna
Provides-Extra: all
# Internationalized Domain Names in Applications (IDNA)
Support for [Internationalized Domain Names in Applications
(IDNA)](https://tools.ietf.org/html/rfc5891) and [Unicode IDNA
Compatibility Processing](https://unicode.org/reports/tr46/). It
supersedes the standard library's `encodings.idna`, which only
implements the 2003 specification, offering broader script coverage and
limiting domains with known security vulnerabilities.
## Usage
Package may be installed from [PyPI](https://pypi.org/project/idna/) via
the typical methods (e.g. `python3 -m pip install idna`)
For typical usage, the `encode` and `decode` functions will take a
domain name argument and perform a conversion to ASCII-compatible encoding
(known as A-labels), or to Unicode strings (known as U-labels)
respectively.
```pycon
>>> import idna
>>> idna.encode('ドメイン.テスト')
b'xn--eckwd4c7c.xn--zckzah'
>>> print(idna.decode('xn--eckwd4c7c.xn--zckzah'))
ドメイン.テスト
```
Conversions can be applied at a per-label basis using the `ulabel` or
`alabel` functions for specialized use cases.
### Compatibility Mapping (UTS #46)
This library provides support for [Unicode IDNA Compatibility
Processing](https://unicode.org/reports/tr46/) which normalizes input from
different potential ways a user may input a domain prior to performing the IDNA
conversion operations. This functionality, known as a
[mapping](https://tools.ietf.org/html/rfc5895), is considered by the
specification to be a local user-interface issue distinct from IDNA
conversion functionality.
For example, "Königsgäßchen" is not a permissible label as capital letters
are not allowed. UTS 46 will convert this into lower case prior to applying
the IDNA conversion.
```pycon
>>> import idna
>>> idna.encode('Königsgäßchen')
...
idna.core.InvalidCodepoint: Codepoint U+004B at position 1 of 'Königsgäßchen' not allowed
>>> idna.encode('Königsgäßchen', uts46=True)
b'xn--knigsgchen-b4a3dun'
>>> idna.decode('xn--knigsgchen-b4a3dun')
'königsgäßchen'
```
When performing a decode operation for display purposes, `decode()`
accepts a `display=True` argument that leaves any `xn--` label that
fails to decode unchanged. This is useful for user interface display
where a domain is in use, the A-label form can be presented when it
is not a valid IDN.
## Exceptions
All errors raised during conversion derive from the `idna.IDNAError`
base class. The more specific exceptions are:
* `idna.IDNABidiError` — raised when a label contains an illegal
combination of left-to-right and right-to-left characters.
* `idna.InvalidCodepoint` — raised when a label contains a codepoint
that is INVALID for IDNA.
* `idna.InvalidCodepointContext` — raised when a CONTEXTO or CONTEXTJ
codepoint appears in a position whose contextual requirements are
not satisfied.
## Command-line tool
The package supports command-line usage to convert domain names
between their Unicode and ASCII-compatible forms. It can be run either
as a module (`python3 -m idna`) or, once installed (such as with `uv
tool` or `pipx`), via the `idna` script:
```bash
$ uv tool install idna
$ idna xn--e1afmkfd.xn--p1ai
пример.рф
$ idna пример.рф
xn--e1afmkfd.xn--p1ai
```
Mode can be specified with `-e`/`--encode` or `-d`/`--decode`, otherwise
it will be chosen automatically based on the first input. Multiple
domains can be supplied either as arguments or through standard input.
UTS #46 mapping is applied by default, which lets the tool accept
inputs that aren't strictly valid IDNA 2008 by normalising them first,
pass `--strict` to disable UTS #46.
Conversion failures are reported on stderr together with the
offending input; processing continues with the remaining domains and
the tool exits with a non-zero status if any conversion failed.
## Additional Notes
* **Version support**. This library supports Python 3.9 and higher.
As this library serves as a low-level toolkit for a variety of
applications, we strive to support all versions of Python that are
not beyond end-of-life.
* **Emoji**. It is an occasional request to support emoji domains in
this library. Encoding of symbols like emoji is expressly prohibited by
the IDNA technical standard, and emoji domains are broadly phased
out across the domain industry due to associated security risks.
* **Regenerating lookup tables**. The IDNA and UTS 46 functionality
relies upon pre-calculated lookup tables, generated using the
`idna-data` script in [`tools/`](tools/README.md).

28
.venv/lib/python3.12/site-packages/idna-3.18.dist-info/RECORD Normal file
View File

@@ -0,0 +1,28 @@
../../../bin/idna,sha256=3wuhjE7EPKBb8A9yI8YbMrjTMugSR9o3GN-ul65e2ig,245
idna-3.18.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
idna-3.18.dist-info/METADATA,sha256=Rt_m5axGLQ9oDs2avPZugptqIzSCS02eOXmzETXK8oE,6119
idna-3.18.dist-info/RECORD,,
idna-3.18.dist-info/WHEEL,sha256=G2gURzTEtmeR8nrdXUJfNiB3VYVxigPQ-bEQujpNiNs,82
idna-3.18.dist-info/entry_points.txt,sha256=7H3nGOHap3jnLE5e7q7Ywr9Vq8axB7WIj5-C_4N2vhw,38
idna-3.18.dist-info/licenses/LICENSE.md,sha256=GppPDj1HmickDd1ZqRN6ZqtKD539yMphiMwL_YUYfwQ,1541
idna/__init__.py,sha256=MPqNDLZbXqGaNdXxAFhiqFPKEQXju2jNQhCey6-5eJM,868
idna/__main__.py,sha256=4JMK66Wj4uLZTKbF-sT3LAxOsr6buig77PmOkJCRRxw,83
idna/__pycache__/__init__.cpython-312.pyc,,
idna/__pycache__/__main__.cpython-312.pyc,,
idna/__pycache__/cli.cpython-312.pyc,,
idna/__pycache__/codec.cpython-312.pyc,,
idna/__pycache__/compat.cpython-312.pyc,,
idna/__pycache__/core.cpython-312.pyc,,
idna/__pycache__/idnadata.cpython-312.pyc,,
idna/__pycache__/intranges.cpython-312.pyc,,
idna/__pycache__/package_data.cpython-312.pyc,,
idna/__pycache__/uts46data.cpython-312.pyc,,
idna/cli.py,sha256=swqJLMNc8Uzs60KziNpbWnHuqlG3WRQwJSbo4n8xDAo,4139
idna/codec.py,sha256=JRbo-f7pEkLdWeiH89Z72UR4VBYhvKDFrQBeNX6sRDE,5040
idna/compat.py,sha256=AepA39ceRHxkfHP41-FvKW5Ki-f4PfUZ90RUMlCNdmo,1353
idna/core.py,sha256=SfOr1xO3PoE0RDYx7bMciAnjiyjJPbPw_93AB5IUYOw,24685
idna/idnadata.py,sha256=Af-mo8WBmkhAK6TyXKOQH88OX0mQNDKtdL7UWtQpppk,44862
idna/intranges.py,sha256=g49scLSkqJtAhLmOODa7hVHriSjmb60tiTsEoocJdBI,1851
idna/package_data.py,sha256=TeI94EqAFAFaXfBJwsOPUMLn2969uirPa-DaeaceAyU,21
idna/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
idna/uts46data.py,sha256=jujNz5QqWMcJf-XYLv4X1jBvb5FlI0t6-e1mILsgbPk,234325

4
.venv/lib/python3.12/site-packages/idna-3.18.dist-info/WHEEL Normal file
View File

@@ -0,0 +1,4 @@
Wheel-Version: 1.0
Generator: flit 3.12.0
Root-Is-Purelib: true
Tag: py3-none-any

3
.venv/lib/python3.12/site-packages/idna-3.18.dist-info/entry_points.txt Normal file
View File

@@ -0,0 +1,3 @@
[console_scripts]
idna=idna.cli:main

31
.venv/lib/python3.12/site-packages/idna-3.18.dist-info/licenses/LICENSE.md Normal file
View File

@@ -0,0 +1,31 @@
BSD 3-Clause License
Copyright (c) 2013-2026, Kim Davies and contributors.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

45
.venv/lib/python3.12/site-packages/idna/__init__.py Normal file
View File

@@ -0,0 +1,45 @@
from .core import (
IDNABidiError,
IDNAError,
InvalidCodepoint,
InvalidCodepointContext,
alabel,
check_bidi,
check_hyphen_ok,
check_initial_combiner,
check_label,
check_nfc,
decode,
encode,
ulabel,
uts46_remap,
valid_contextj,
valid_contexto,
valid_label_length,
valid_string_length,
)
from .intranges import intranges_contain
from .package_data import __version__
__all__ = [
"__version__",
"IDNABidiError",
"IDNAError",
"InvalidCodepoint",
"InvalidCodepointContext",
"alabel",
"check_bidi",
"check_hyphen_ok",
"check_initial_combiner",
"check_label",
"check_nfc",
"decode",
"encode",
"intranges_contain",
"ulabel",
"uts46_remap",
"valid_contextj",
"valid_contexto",
"valid_label_length",
"valid_string_length",
]

6
.venv/lib/python3.12/site-packages/idna/__main__.py Normal file
View File

@@ -0,0 +1,6 @@
import sys
from .cli import main
if __name__ == "__main__":
sys.exit(main())

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/__init__.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/__main__.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/cli.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/codec.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/compat.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/core.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/idnadata.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/intranges.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/package_data.cpython-312.pyc Normal file
View File

Binary file not shown.

BIN
.venv/lib/python3.12/site-packages/idna/__pycache__/uts46data.cpython-312.pyc Normal file
View File

Binary file not shown.

128
.venv/lib/python3.12/site-packages/idna/cli.py Normal file
View File

@@ -0,0 +1,128 @@
"""Command-line interface for the :mod:`idna` package.
Invoked via ``python -m idna``. See :func:`main` for the entry point.
"""
import argparse
import sys
from collections.abc import Iterable
from itertools import chain
from typing import IO, Optional
from . import IDNAError, decode, encode
from .core import _alabel_prefix, _unicode_dots_re
from .package_data import __version__
def _looks_like_alabel(s: str) -> bool:
"""Return True if any label in ``s`` carries the ``xn--`` ACE prefix."""
prefix = _alabel_prefix.decode("ascii")
return any(label.lower().startswith(prefix) for label in _unicode_dots_re.split(s))
def _build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="python -m idna",
description=(
"Convert a domain name between its Unicode (U-label) and "
"ASCII-compatible (A-label) forms. With no mode flag, the "
"direction is chosen from the first input — if it contains "
"an xn-- label the stream is decoded, otherwise it is "
"encoded — and the same mode is applied to every remaining "
"input. UTS #46 mapping is applied by default; pass "
"--strict to disable it. When no domains are given on the "
"command line and stdin is piped, one domain per line is "
"read from stdin."
),
)
mode = parser.add_mutually_exclusive_group()
mode.add_argument(
"-e",
"--encode",
dest="mode",
action="store_const",
const="encode",
help="Encode the input to its ASCII A-label form.",
)
mode.add_argument(
"-d",
"--decode",
dest="mode",
action="store_const",
const="decode",
help="Decode the input from its ASCII A-label form.",
)
parser.add_argument(
"--strict",
action="store_true",
help="Disable the default UTS #46 mapping and apply IDNA 2008 rules verbatim.",
)
parser.add_argument(
"--version",
action="version",
version=f"idna {__version__}",
)
parser.add_argument(
"domain",
nargs="*",
help="One or more domain names to convert. Omit to read from stdin.",
)
return parser
def _iter_stdin(stream: IO[str]) -> Iterable[str]:
"""Yield non-empty stripped lines from ``stream``, ignoring blanks."""
for line in stream:
stripped = line.strip()
if stripped:
yield stripped
def _convert_one(domain: str, mode: str, uts46: bool) -> bool:
"""Convert ``domain`` and write the result; return ``False`` on failure."""
try:
if mode == "decode":
print(decode(domain, uts46=uts46))
else:
print(encode(domain, uts46=uts46).decode("ascii"))
except IDNAError as err:
print(f"idna: {mode} failed for {domain!r}: {err}", file=sys.stderr)
return False
return True
def main(argv: Optional[list[str]] = None) -> int:
"""Entry point for ``python -m idna``.
When more than one domain is supplied (via positional arguments or
piped stdin) and no mode flag is given, the first input determines
the direction and that mode is applied uniformly to the rest.
:param argv: Argument list excluding the program name. Defaults to
:data:`sys.argv` when ``None``.
:returns: ``0`` on success, ``1`` if any conversion fails.
"""
parser = _build_parser()
args = parser.parse_args(argv)
uts46 = not args.strict
if args.domain:
domains: Iterable[str] = args.domain
elif not sys.stdin.isatty():
domains = _iter_stdin(sys.stdin)
else:
parser.error("a domain argument is required when stdin is a terminal")
iterator = iter(domains)
first = next(iterator, None)
if first is None:
return 0
mode = args.mode or ("decode" if _looks_like_alabel(first) else "encode")
results = [_convert_one(domain, mode, uts46) for domain in chain([first], iterator)]
return 0 if all(results) else 1
if __name__ == "__main__":
sys.exit(main())

159
.venv/lib/python3.12/site-packages/idna/codec.py Normal file
View File

@@ -0,0 +1,159 @@
import codecs
from typing import Any, Optional
from .core import IDNAError, _unicode_dots_re, alabel, decode, encode, ulabel
class Codec(codecs.Codec):
"""Stateless IDNA 2008 codec.
Implements the :class:`codecs.Codec` protocol so that the whole-domain
encoder (:func:`idna.encode`) and decoder (:func:`idna.decode`) are
accessible through the standard codec machinery as ``"idna2008"``.
Only the ``"strict"`` error handler is supported; any other handler
raises :exc:`~idna.IDNAError`.
"""
def encode(self, data: str, errors: str = "strict") -> tuple[bytes, int]: # ty: ignore[invalid-method-override]
if errors != "strict":
raise IDNAError(f'Unsupported error handling "{errors}"')
if not data:
return b"", 0
return encode(data), len(data)
def decode(self, data: bytes, errors: str = "strict") -> tuple[str, int]: # ty: ignore[invalid-method-override]
if errors != "strict":
raise IDNAError(f'Unsupported error handling "{errors}"')
if not data:
return "", 0
return decode(data), len(data)
class IncrementalEncoder(codecs.BufferedIncrementalEncoder):
"""Incremental IDNA 2008 encoder.
Buffers a partial trailing label across calls until either the next
label separator is seen or ``final=True``, so that streamed input is
encoded one whole label at a time. Any of the four Unicode label
separators (``U+002E``, ``U+3002``, ``U+FF0E``, ``U+FF61``) ends a
label; the result always uses ``U+002E`` as the separator.
Only the ``"strict"`` error handler is supported.
"""
def _buffer_encode(self, data: str, errors: str, final: bool) -> tuple[bytes, int]: # ty: ignore[invalid-method-override]
if errors != "strict":
raise IDNAError(f'Unsupported error handling "{errors}"')
if not data:
return b"", 0
labels = _unicode_dots_re.split(data)
trailing_dot = b""
if labels:
if not labels[-1]:
trailing_dot = b"."
del labels[-1]
elif not final:
# Keep potentially unfinished label until the next call
del labels[-1]
if labels:
trailing_dot = b"."
result = []
size = 0
for label in labels:
result.append(alabel(label))
if size:
size += 1
size += len(label)
# Join with U+002E
result_bytes = b".".join(result) + trailing_dot
size += len(trailing_dot)
return result_bytes, size
class IncrementalDecoder(codecs.BufferedIncrementalDecoder):
"""Incremental IDNA 2008 decoder.
Buffers a partial trailing label across calls until either the next
label separator is seen or ``final=True``, so that streamed input is
decoded one whole label at a time.
Only the ``"strict"`` error handler is supported.
"""
def _buffer_decode(self, data: Any, errors: str, final: bool) -> tuple[str, int]: # ty: ignore[invalid-method-override]
if errors != "strict":
raise IDNAError(f'Unsupported error handling "{errors}"')
if not data:
return ("", 0)
if not isinstance(data, str):
data = str(data, "ascii")
labels = _unicode_dots_re.split(data)
trailing_dot = ""
if labels:
if not labels[-1]:
trailing_dot = "."
del labels[-1]
elif not final:
# Keep potentially unfinished label until the next call
del labels[-1]
if labels:
trailing_dot = "."
result = []
size = 0
for label in labels:
result.append(ulabel(label))
if size:
size += 1
size += len(label)
result_str = ".".join(result) + trailing_dot
size += len(trailing_dot)
return (result_str, size)
class StreamWriter(Codec, codecs.StreamWriter):
pass
class StreamReader(Codec, codecs.StreamReader):
pass
def search_function(name: str) -> Optional[codecs.CodecInfo]:
"""Codec search function registered with :mod:`codecs`.
Returns a :class:`codecs.CodecInfo` for the ``"idna2008"`` codec name
so that ``str.encode("idna2008")`` and ``bytes.decode("idna2008")``
invoke the IDNA 2008 codec defined in this module.
:param name: The codec name being looked up.
:returns: A :class:`codecs.CodecInfo` instance if ``name`` is
``"idna2008"``, otherwise ``None``.
"""
if name != "idna2008":
return None
return codecs.CodecInfo(
name=name,
encode=Codec().encode,
decode=Codec().decode, # type: ignore
incrementalencoder=IncrementalEncoder,
incrementaldecoder=IncrementalDecoder,
streamwriter=StreamWriter,
streamreader=StreamReader,
)
codecs.register(search_function)

41
.venv/lib/python3.12/site-packages/idna/compat.py Normal file
View File

@@ -0,0 +1,41 @@
from typing import Any, Union
from .core import decode, encode
def ToASCII(label: str) -> bytes:
"""Compatibility shim for :rfc:`3490` ``ToASCII``.
Delegates to :func:`idna.encode` (IDNA 2008). Provided to ease porting
of code written against the legacy :mod:`encodings.idna` API; new code
should call :func:`idna.encode` directly.
:param label: The label or domain to encode.
:returns: The encoded form as ASCII :class:`bytes`.
"""
return encode(label)
def ToUnicode(label: Union[bytes, bytearray]) -> str:
"""Compatibility shim for :rfc:`3490` ``ToUnicode``.
Delegates to :func:`idna.decode` (IDNA 2008). Provided to ease porting
of code written against the legacy :mod:`encodings.idna` API; new code
should call :func:`idna.decode` directly.
:param label: The label or domain to decode.
:returns: The decoded Unicode form.
"""
return decode(label)
def nameprep(s: Any) -> None:
"""Stub for :rfc:`3491` Nameprep, which is not used by IDNA 2008.
IDNA 2008 (:rfc:`5891`) replaces Nameprep with the per-codepoint
validity classes from :rfc:`5892`; this function exists only to
return a clear error if legacy code attempts to call it.
:raises NotImplementedError: Always.
"""
raise NotImplementedError("IDNA 2008 does not utilise nameprep protocol")

648
.venv/lib/python3.12/site-packages/idna/core.py Normal file
View File

@@ -0,0 +1,648 @@
import bisect
import re
import unicodedata
import warnings
from typing import Optional, Union
from . import idnadata
from .intranges import intranges_contain
_virama_combining_class = 9
_alabel_prefix = b"xn--"
_max_input_length = 1024
_unicode_dots_re = re.compile("[\u002e\u3002\uff0e\uff61]")
# Bidi category sets from RFC 5893, hoisted out of the per-codepoint loop
_bidi_rtl_first = frozenset({"R", "AL"})
_bidi_rtl_categories = frozenset({"R", "AL", "AN"})
_bidi_rtl_allowed = frozenset({"R", "AL", "AN", "EN", "ES", "CS", "ET", "ON", "BN", "NSM"})
_bidi_rtl_valid_ending = frozenset({"R", "AL", "EN", "AN"})
_bidi_rtl_numeric = frozenset({"AN", "EN"})
_bidi_ltr_allowed = frozenset({"L", "EN", "ES", "CS", "ET", "ON", "BN", "NSM"})
_bidi_ltr_valid_ending = frozenset({"L", "EN"})
_bidi_joiner_l_or_d = frozenset({"L", "D"})
_bidi_joiner_r_or_d = frozenset({"R", "D"})
def _joining_type(cp: int) -> Optional[str]:
for jt, ranges in idnadata.joining_types.items():
if intranges_contain(cp, ranges):
return jt
return None
class IDNAError(UnicodeError):
"""Base exception for all IDNA-encoding related problems"""
class IDNABidiError(IDNAError):
"""Exception when bidirectional requirements are not satisfied"""
class InvalidCodepoint(IDNAError):
"""Exception when a disallowed or unallocated codepoint is used"""
class InvalidCodepointContext(IDNAError):
"""Exception when the codepoint is not valid in the context it is used"""
def _combining_class(cp: int) -> int:
v = unicodedata.combining(chr(cp))
if v == 0 and not unicodedata.name(chr(cp)):
raise ValueError("Unknown character in unicodedata")
return v
def _is_script(cp: str, script: str) -> bool:
return intranges_contain(ord(cp), idnadata.scripts[script])
def _punycode(s: str) -> bytes:
return s.encode("punycode")
def _unot(s: int) -> str:
return f"U+{s:04X}"
def valid_label_length(label: Union[bytes, str]) -> bool:
"""Check that a label does not exceed the maximum permitted length.
Per :rfc:`1035` (and :rfc:`5891` §4.2.4) a DNS label must not exceed
63 octets. The argument may be either a :class:`str` (a U-label, where
length is measured in characters) or :class:`bytes` (an A-label, where
length is measured in octets).
:param label: The label to check.
:returns: ``True`` if the label is within the length limit, otherwise
``False``.
"""
return len(label) <= 63
def valid_string_length(domain: Union[bytes, str], trailing_dot: bool) -> bool:
"""Check that a full domain name does not exceed the maximum length.
Per :rfc:`1035`, a domain name is limited to 253 octets when no trailing
dot is present, or 254 octets when one is included.
:param domain: The full (possibly multi-label) domain name.
:param trailing_dot: ``True`` if ``domain`` includes a trailing ``.``.
:returns: ``True`` if the domain is within the length limit, otherwise
``False``.
"""
return len(domain) <= (254 if trailing_dot else 253)
def check_bidi(label: str, check_ltr: bool = False) -> bool:
"""Validate the Bidi Rule from :rfc:`5893` for a single label.
The Bidi Rule constrains how bidirectional characters (Hebrew, Arabic,
etc.) may appear within a label. By default the check is only applied
when the label contains at least one right-to-left character (Unicode
bidirectional categories ``R``, ``AL``, or ``AN``); set ``check_ltr``
to ``True`` to apply it to LTR-only labels as well.
:param label: The label to validate, as a Unicode string.
:param check_ltr: If ``True``, apply the rules even when the label
contains no RTL characters.
:returns: ``True`` if the label satisfies the Bidi Rule.
:raises IDNABidiError: If any of Bidi Rule conditions 1-6 are violated,
or if the directional category of a codepoint cannot be determined.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long")
# Bidi rules should only be applied if string contains RTL characters
bidi_label = False
for idx, cp in enumerate(label, 1):
direction = unicodedata.bidirectional(cp)
if direction == "":
# String likely comes from a newer version of Unicode
raise IDNABidiError(f"Unknown directionality in label {label!r} at position {idx}")
if direction in _bidi_rtl_categories:
bidi_label = True
if not bidi_label and not check_ltr:
return True
# Bidi rule 1
direction = unicodedata.bidirectional(label[0])
if direction in _bidi_rtl_first:
rtl = True
elif direction == "L":
rtl = False
else:
raise IDNABidiError(f"First codepoint in label {label!r} must be directionality L, R or AL")
valid_ending = False
number_type: Optional[str] = None
for idx, cp in enumerate(label, 1):
direction = unicodedata.bidirectional(cp)
if rtl:
# Bidi rule 2
if direction not in _bidi_rtl_allowed:
raise IDNABidiError(f"Invalid direction for codepoint at position {idx} in a right-to-left label")
# Bidi rule 3
if direction in _bidi_rtl_valid_ending:
valid_ending = True
elif direction != "NSM":
valid_ending = False
# Bidi rule 4
if direction in _bidi_rtl_numeric:
if not number_type:
number_type = direction
elif number_type != direction:
raise IDNABidiError("Can not mix numeral types in a right-to-left label")
else:
# Bidi rule 5
if direction not in _bidi_ltr_allowed:
raise IDNABidiError(f"Invalid direction for codepoint at position {idx} in a left-to-right label")
# Bidi rule 6
if direction in _bidi_ltr_valid_ending:
valid_ending = True
elif direction != "NSM":
valid_ending = False
if not valid_ending:
raise IDNABidiError("Label ends with illegal codepoint directionality")
return True
def check_initial_combiner(label: str) -> bool:
"""Reject labels that begin with a combining mark.
Per :rfc:`5891` §4.2.3.2 a label must not start with a character of
Unicode general category ``M`` (Mark).
:param label: The label to check.
:returns: ``True`` if the first character is not a combining mark.
:raises IDNAError: If the label begins with a combining character.
"""
if unicodedata.category(label[0])[0] == "M":
raise IDNAError("Label begins with an illegal combining character")
return True
def check_hyphen_ok(label: str) -> bool:
"""Validate the hyphen restrictions for a label.
Per :rfc:`5891` §4.2.3.1 a label must not start or end with a hyphen
(``U+002D``), and must not have hyphens in both the third and fourth
positions (the prefix reserved for A-labels).
:param label: The label to check.
:returns: ``True`` if the hyphen restrictions are satisfied.
:raises IDNAError: If any of the hyphen restrictions are violated.
"""
if label[2:4] == "--":
raise IDNAError("Label has disallowed hyphens in 3rd and 4th position")
if label[0] == "-" or label[-1] == "-":
raise IDNAError("Label must not start or end with a hyphen")
return True
def check_nfc(label: str) -> None:
"""Require that a label is in Unicode Normalization Form C.
:param label: The label to check.
:raises IDNAError: If ``label`` differs from its NFC normalisation.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long")
if unicodedata.normalize("NFC", label) != label:
raise IDNAError("Label must be in Normalization Form C")
def valid_contextj(label: str, pos: int) -> bool:
"""Validate the CONTEXTJ rules from :rfc:`5892` Appendix A.
These rules govern the contextual use of the joiner codepoints
``U+200C`` (ZERO WIDTH NON-JOINER, Appendix A.1) and ``U+200D``
(ZERO WIDTH JOINER, Appendix A.2) within a label.
:param label: The label containing the codepoint.
:param pos: Index of the joiner codepoint within ``label``.
:returns: ``True`` if the codepoint at ``pos`` satisfies its CONTEXTJ
rule, ``False`` otherwise (including when the codepoint at
``pos`` is not a recognised joiner).
:raises ValueError: If an adjacent codepoint has no Unicode name when
determining its combining class.
:raises IDNAError: If ``label`` exceeds the defensive input length limit.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long")
cp_value = ord(label[pos])
if cp_value == 0x200C:
if pos > 0 and _combining_class(ord(label[pos - 1])) == _virama_combining_class:
return True
ok = False
for i in range(pos - 1, -1, -1):
joining_type = _joining_type(ord(label[i]))
if joining_type == "T":
continue
if joining_type in _bidi_joiner_l_or_d:
ok = True
break
break
if not ok:
return False
ok = False
for i in range(pos + 1, len(label)):
joining_type = _joining_type(ord(label[i]))
if joining_type == "T":
continue
if joining_type in _bidi_joiner_r_or_d:
ok = True
break
break
return ok
if cp_value == 0x200D:
return pos > 0 and _combining_class(ord(label[pos - 1])) == _virama_combining_class
return False
def valid_contexto(label: str, pos: int, exception: bool = False) -> bool:
"""Validate the CONTEXTO rules from :rfc:`5892` Appendix A.
Covers the contextual rules for codepoints such as MIDDLE DOT
(``U+00B7``), Greek lower numeral sign, Hebrew punctuation, Katakana
middle dot, and the Arabic-Indic / Extended Arabic-Indic digit ranges.
:param label: The label containing the codepoint.
:param pos: Index of the codepoint within ``label``.
:param exception: Reserved for forward compatibility; currently unused.
:returns: ``True`` if the codepoint at ``pos`` satisfies its CONTEXTO
rule, ``False`` otherwise (including when the codepoint is not a
recognised CONTEXTO codepoint).
:raises IDNAError: If ``label`` exceeds the defensive input length limit.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long")
cp_value = ord(label[pos])
if cp_value == 0x00B7:
return 0 < pos < len(label) - 1 and ord(label[pos - 1]) == 0x006C and ord(label[pos + 1]) == 0x006C
if cp_value == 0x0375:
if pos < len(label) - 1 and len(label) > 1:
return _is_script(label[pos + 1], "Greek")
return False
if cp_value in {0x05F3, 0x05F4}:
if pos > 0:
return _is_script(label[pos - 1], "Hebrew")
return False
if cp_value == 0x30FB:
for cp in label:
if cp == "\u30fb":
continue
if _is_script(cp, "Hiragana") or _is_script(cp, "Katakana") or _is_script(cp, "Han"):
return True
return False
if 0x660 <= cp_value <= 0x669:
return not any(0x6F0 <= ord(cp) <= 0x06F9 for cp in label)
if 0x6F0 <= cp_value <= 0x6F9:
return not any(0x660 <= ord(cp) <= 0x0669 for cp in label)
return False
def check_label(label: Union[str, bytes, bytearray]) -> None:
"""Run the full set of IDNA 2008 validity checks on a single label.
Applies, in order: NFC normalisation (:func:`check_nfc`), hyphen
restrictions (:func:`check_hyphen_ok`), the no-leading-combiner rule
(:func:`check_initial_combiner`), per-codepoint validity (PVALID,
CONTEXTJ, CONTEXTO classes from :rfc:`5892`), and the Bidi Rule
(:func:`check_bidi`).
:param label: The label to validate. ``bytes`` or ``bytearray`` input
is decoded as UTF-8 first.
:raises IDNAError: If the label is empty or fails a structural rule.
:raises InvalidCodepoint: If the label contains a DISALLOWED or
UNASSIGNED codepoint.
:raises InvalidCodepointContext: If a CONTEXTJ or CONTEXTO codepoint
is not valid in its context.
:raises IDNABidiError: If the Bidi Rule is violated.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long")
if isinstance(label, (bytes, bytearray)):
label = label.decode("utf-8")
if len(label) == 0:
raise IDNAError("Empty Label")
# Reject on domain length rather than label length so support some UTS 46
# use cases, still reducing processing of label contextual rules
if not valid_string_length(label, trailing_dot=True):
raise IDNAError("Label too long")
check_nfc(label)
check_hyphen_ok(label)
check_initial_combiner(label)
for pos, cp in enumerate(label):
cp_value = ord(cp)
if intranges_contain(cp_value, idnadata.codepoint_classes["PVALID"]):
continue
if intranges_contain(cp_value, idnadata.codepoint_classes["CONTEXTJ"]):
try:
if not valid_contextj(label, pos):
raise InvalidCodepointContext(f"Joiner {_unot(cp_value)} not allowed at position {pos + 1} in {label!r}")
except ValueError as err:
raise IDNAError(
f"Unknown codepoint adjacent to joiner {_unot(cp_value)} at position {pos + 1} in {label!r}"
) from err
elif intranges_contain(cp_value, idnadata.codepoint_classes["CONTEXTO"]):
if not valid_contexto(label, pos):
raise InvalidCodepointContext(f"Codepoint {_unot(cp_value)} not allowed at position {pos + 1} in {label!r}")
else:
raise InvalidCodepoint(f"Codepoint {_unot(cp_value)} at position {pos + 1} of {label!r} not allowed")
check_bidi(label)
def alabel(label: str) -> bytes:
"""Convert a single U-label into its A-label form.
The result is the ASCII-Compatible Encoding (ACE) form per :rfc:`5891`
§4: the label is validated, Punycode-encoded, and prefixed with
``xn--``. Pure ASCII labels that are already valid IDNA labels are
returned unchanged (as :class:`bytes`).
:param label: The label to convert, as a Unicode string.
:returns: The A-label as ASCII-encoded :class:`bytes`.
:raises IDNAError: If the label is invalid or the resulting A-label
exceeds 63 octets.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long")
try:
label_bytes = label.encode("ascii")
except UnicodeEncodeError:
pass
else:
ulabel(label_bytes)
if not valid_label_length(label_bytes):
raise IDNAError("Label too long")
return label_bytes
check_label(label)
label_bytes = _alabel_prefix + _punycode(label)
if not valid_label_length(label_bytes):
raise IDNAError("Label too long")
return label_bytes
def ulabel(label: Union[str, bytes, bytearray]) -> str:
"""Convert a single A-label into its U-label form.
Performs the inverse of :func:`alabel`: an ``xn--``-prefixed label is
Punycode-decoded and validated. Labels that are already Unicode (or
plain ASCII without the ACE prefix) are validated and returned as a
Unicode string.
:param label: The label to convert. ``bytes`` or ``bytearray`` input
is treated as ASCII.
:returns: The U-label as a Unicode string.
:raises IDNAError: If the label is malformed or fails validation.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long")
if not isinstance(label, (bytes, bytearray)):
try:
label_bytes = label.encode("ascii")
except UnicodeEncodeError:
check_label(label)
return label
else:
label_bytes = bytes(label)
label_bytes = label_bytes.lower()
if label_bytes.startswith(_alabel_prefix):
label_bytes = label_bytes[len(_alabel_prefix) :]
if not label_bytes:
raise IDNAError("Malformed A-label, no Punycode eligible content found")
if label_bytes.endswith(b"-"):
raise IDNAError("A-label must not end with a hyphen")
else:
check_label(label_bytes)
return label_bytes.decode("ascii")
try:
label = label_bytes.decode("punycode")
except UnicodeError as err:
raise IDNAError("Invalid A-label") from err
check_label(label)
return label
def uts46_remap(domain: str, std3_rules: bool = True, transitional: bool = False) -> str:
"""Apply the UTS #46 character mapping to a domain string.
Implements the mapping table from `UTS #46 §4
<https://www.unicode.org/reports/tr46/>`_: each character is kept,
replaced, or rejected based on its status (``V``, ``M``, ``D``, ``3``,
``I``). The result is returned in Normalisation Form C.
:param domain: The full domain name to remap.
:param std3_rules: If ``True``, apply the stricter STD3 ASCII rules
(status ``3`` codepoints raise instead of being kept or mapped).
:param transitional: If ``True``, use transitional processing (status
``D`` codepoints are mapped instead of kept). Transitional
processing has been removed from UTS #46 and this option is
retained only for backwards compatibility.
:returns: The remapped domain, in Normalisation Form C.
:raises InvalidCodepoint: If the domain contains a disallowed
codepoint under the chosen rules.
:raises IDNAError: If ``domain`` exceeds the defensive input length limit.
"""
if len(domain) > _max_input_length:
raise IDNAError("Domain too long")
from .uts46data import uts46_replacements, uts46_starts, uts46_statuses
output = ""
for pos, char in enumerate(domain):
code_point = ord(char)
i = code_point if code_point < 256 else bisect.bisect_right(uts46_starts, code_point) - 1
status = chr(uts46_statuses[i])
replacement: Optional[str] = uts46_replacements[i]
# UTS #46 §4: V is always valid, D is deviation (kept unless transitional),
# 3 is disallowed-STD3 (kept unmapped if std3_rules is off and no mapping).
keep_as_is = (
status == "V" or (status == "D" and not transitional) or (status == "3" and not std3_rules and replacement is None)
)
# M is mapped, 3-with-replacement and transitional D fall through to the
# same replacement output path.
use_replacement = replacement is not None and (
status == "M" or (status == "3" and not std3_rules) or (status == "D" and transitional)
)
if keep_as_is:
output += char
elif use_replacement:
assert replacement is not None # narrowed by use_replacement
output += replacement
elif status == "I":
continue
else:
raise InvalidCodepoint(f"Codepoint {_unot(code_point)} not allowed at position {pos + 1} in {domain!r}")
return unicodedata.normalize("NFC", output)
def encode(
s: Union[str, bytes, bytearray],
strict: bool = False,
uts46: bool = False,
std3_rules: bool = False,
transitional: bool = False,
) -> bytes:
"""Encode a Unicode domain name into its ASCII (A-label) form.
Splits the input on label separators (only ``U+002E`` if ``strict`` is
set; otherwise also IDEOGRAPHIC FULL STOP ``U+3002``, FULLWIDTH FULL
STOP ``U+FF0E``, and HALFWIDTH IDEOGRAPHIC FULL STOP ``U+FF61``),
encodes each label with :func:`alabel`, and rejoins them with ``.``.
Optionally pre-processes the input through :func:`uts46_remap`.
:param s: The domain name to encode.
:param strict: If ``True``, only ``U+002E`` is recognised as a label
separator.
:param uts46: If ``True``, apply UTS #46 mapping before encoding.
:param std3_rules: Forwarded to :func:`uts46_remap` when ``uts46`` is
``True``.
:param transitional: Forwarded to :func:`uts46_remap` when ``uts46``
is ``True``. Deprecated: emits a :class:`DeprecationWarning` and
will be removed in a future version.
:returns: The encoded domain as ASCII :class:`bytes`.
:raises IDNAError: If the domain is empty, contains an invalid label,
or exceeds the maximum domain length.
"""
if transitional:
warnings.warn(
"Transitional processing has been removed from UTS #46. "
"The transitional argument will be removed in a future version.",
DeprecationWarning,
stacklevel=2,
)
if not isinstance(s, str):
try:
s = str(s, "ascii")
except (UnicodeDecodeError, TypeError) as err:
raise IDNAError("should pass a unicode string to the function rather than a byte string.") from err
if len(s) > _max_input_length:
raise IDNAError("Domain too long")
if uts46:
s = uts46_remap(s, std3_rules, transitional)
# Reject inputs that exceed the maximum DNS domain length up-front
# to avoid expensive computation on long inputs.
if not valid_string_length(s, trailing_dot=True):
raise IDNAError("Domain too long")
trailing_dot = False
result = []
labels = s.split(".") if strict else _unicode_dots_re.split(s)
if not labels or labels == [""]:
raise IDNAError("Empty domain")
if labels[-1] == "":
del labels[-1]
trailing_dot = True
for label in labels:
s = alabel(label)
if s:
result.append(s)
else:
raise IDNAError("Empty label")
if trailing_dot:
result.append(b"")
s = b".".join(result)
if not valid_string_length(s, trailing_dot):
raise IDNAError("Domain too long")
return s
def decode(
s: Union[str, bytes, bytearray],
strict: bool = False,
uts46: bool = False,
std3_rules: bool = False,
display: bool = False,
) -> str:
"""Decode an A-label-encoded domain name back to Unicode.
Splits the input on label separators (see :func:`encode` for the
rules), decodes each label with :func:`ulabel`, and rejoins them
with ``.``. Optionally pre-processes the input through
:func:`uts46_remap`.
:param s: The domain name to decode.
:param strict: If ``True``, only ``U+002E`` is recognised as a label
separator.
:param uts46: If ``True``, apply UTS #46 mapping before decoding.
:param std3_rules: Forwarded to :func:`uts46_remap` when ``uts46`` is
``True``.
:param display: If ``True``, any ``xn--`` label that fails IDNA
validation is passed through unchanged (lowercased) rather than
aborting the whole call. Intended for "decode for display"
consumers (e.g. URL libraries, HTTP clients) that want to show
the user the label as it appears on the wire when it cannot be
rendered as Unicode. Matches the per-label recovery prescribed
by UTS #46 §4 and the WHATWG URL "domain to Unicode" algorithm.
:returns: The decoded domain as a Unicode string.
:raises IDNAError: If the input is not valid ASCII, contains an
invalid label, or is empty.
"""
if not isinstance(s, str):
try:
s = str(s, "ascii")
except (UnicodeDecodeError, TypeError) as err:
raise IDNAError("Invalid ASCII in A-label") from err
if len(s) > _max_input_length:
raise IDNAError("Domain too long")
if uts46:
s = uts46_remap(s, std3_rules, False)
# Reject inputs that exceed the maximum DNS domain length up-front
# to avoid expensive computation on long inputs.
if not valid_string_length(s, trailing_dot=True):
raise IDNAError("Domain too long")
trailing_dot = False
result = []
labels = s.split(".") if strict else _unicode_dots_re.split(s)
if not labels or labels == [""]:
raise IDNAError("Empty domain")
if not labels[-1]:
del labels[-1]
trailing_dot = True
for label in labels:
try:
u = ulabel(label)
except IDNAError:
if display and label[:4].lower() == "xn--":
u = label.lower()
else:
raise
if u:
result.append(u)
else:
raise IDNAError("Empty label")
if trailing_dot:
result.append("")
return ".".join(result)

1897
.venv/lib/python3.12/site-packages/idna/idnadata.py Normal file
View File

File diff suppressed because it is too large Load Diff

55
.venv/lib/python3.12/site-packages/idna/intranges.py Normal file
View File

@@ -0,0 +1,55 @@
"""
Given a list of integers, made up of (hopefully) a small number of long runs
of consecutive integers, compute a representation of the form
((start1, end1), (start2, end2) ...). Then answer the question "was x present
in the original list?" in time O(log(# runs)).
"""
import bisect
def intranges_from_list(list_: list[int]) -> tuple[int, ...]:
"""Represent a list of integers as a sequence of ranges:
((start_0, end_0), (start_1, end_1), ...), such that the original
integers are exactly those x such that start_i <= x < end_i for some i.
Ranges are encoded as single integers (start << 32 | end), not as tuples.
"""
sorted_list = sorted(list_)
ranges = []
last_write = -1
for i in range(len(sorted_list)):
if i + 1 < len(sorted_list) and sorted_list[i] == sorted_list[i + 1] - 1:
continue
current_range = sorted_list[last_write + 1 : i + 1]
ranges.append(_encode_range(current_range[0], current_range[-1] + 1))
last_write = i
return tuple(ranges)
def _encode_range(start: int, end: int) -> int:
return (start << 32) | end
def _decode_range(r: int) -> tuple[int, int]:
return (r >> 32), (r & ((1 << 32) - 1))
def intranges_contain(int_: int, ranges: tuple[int, ...]) -> bool:
"""Determine if `int_` falls into one of the ranges in `ranges`."""
tuple_ = _encode_range(int_, 0)
pos = bisect.bisect_left(ranges, tuple_)
# we could be immediately ahead of a tuple (start, end)
# with start < int_ <= end
if pos > 0:
left, right = _decode_range(ranges[pos - 1])
if left <= int_ < right:
return True
# or we could be immediately behind a tuple (int_, end)
if pos < len(ranges):
left, _ = _decode_range(ranges[pos])
if left == int_:
return True
return False

1
.venv/lib/python3.12/site-packages/idna/package_data.py Normal file
View File

@@ -0,0 +1 @@
__version__ = "3.18"

0
.venv/lib/python3.12/site-packages/idna/py.typed Normal file
View File

16896
.venv/lib/python3.12/site-packages/idna/uts46data.py Normal file
View File

File diff suppressed because it is too large Load Diff

760
.venv/lib/python3.12/site-packages/pip-24.0.dist-info/AUTHORS.txt Normal file
View File

@@ -0,0 +1,760 @@
@Switch01
A_Rog
Aakanksha Agrawal
Abhinav Sagar
ABHYUDAY PRATAP SINGH
abs51295
AceGentile
Adam Chainz
Adam Tse
Adam Wentz
admin
Adrien Morison
ahayrapetyan
Ahilya
AinsworthK
Akash Srivastava
Alan Yee
Albert Tugushev
Albert-Guan
albertg
Alberto Sottile
Aleks Bunin
Ales Erjavec
Alethea Flowers
Alex Gaynor
Alex Grönholm
Alex Hedges
Alex Loosley
Alex Morega
Alex Stachowiak
Alexander Shtyrov
Alexandre Conrad
Alexey Popravka
Aleš Erjavec
Alli
Ami Fischman
Ananya Maiti
Anatoly Techtonik
Anders Kaseorg
Andre Aguiar
Andreas Lutro
Andrei Geacar
Andrew Gaul
Andrew Shymanel
Andrey Bienkowski
Andrey Bulgakov
Andrés Delfino
Andy Freeland
Andy Kluger
Ani Hayrapetyan
Aniruddha Basak
Anish Tambe
Anrs Hu
Anthony Sottile
Antoine Musso
Anton Ovchinnikov
Anton Patrushev
Antonio Alvarado Hernandez
Antony Lee
Antti Kaihola
Anubhav Patel
Anudit Nagar
Anuj Godase
AQNOUCH Mohammed
AraHaan
Arindam Choudhury
Armin Ronacher
Artem
Arun Babu Neelicattu
Ashley Manton
Ashwin Ramaswami
atse
Atsushi Odagiri
Avinash Karhana
Avner Cohen
Awit (Ah-Wit) Ghirmai
Baptiste Mispelon
Barney Gale
barneygale
Bartek Ogryczak
Bastian Venthur
Ben Bodenmiller
Ben Darnell
Ben Hoyt
Ben Mares
Ben Rosser
Bence Nagy
Benjamin Peterson
Benjamin VanEvery
Benoit Pierre
Berker Peksag
Bernard
Bernard Tyers
Bernardo B. Marques
Bernhard M. Wiedemann
Bertil Hatt
Bhavam Vidyarthi
Blazej Michalik
Bogdan Opanchuk
BorisZZZ
Brad Erickson
Bradley Ayers
Brandon L. Reiss
Brandt Bucher
Brett Randall
Brett Rosen
Brian Cristante
Brian Rosner
briantracy
BrownTruck
Bruno Oliveira
Bruno Renié
Bruno S
Bstrdsmkr
Buck Golemon
burrows
Bussonnier Matthias
bwoodsend
c22
Caleb Martinez
Calvin Smith
Carl Meyer
Carlos Liam
Carol Willing
Carter Thayer
Cass
Chandrasekhar Atina
Chih-Hsuan Yen
Chris Brinker
Chris Hunt
Chris Jerdonek
Chris Kuehl
Chris McDonough
Chris Pawley
Chris Pryer
Chris Wolfe
Christian Clauss
Christian Heimes
Christian Oudard
Christoph Reiter
Christopher Hunt
Christopher Snyder
cjc7373
Clark Boylan
Claudio Jolowicz
Clay McClure
Cody
Cody Soyland
Colin Watson
Collin Anderson
Connor Osborn
Cooper Lees
Cooper Ry Lees
Cory Benfield
Cory Wright
Craig Kerstiens
Cristian Sorinel
Cristina
Cristina Muñoz
Curtis Doty
cytolentino
Daan De Meyer
Dale
Damian
Damian Quiroga
Damian Shaw
Dan Black
Dan Savilonis
Dan Sully
Dane Hillard
daniel
Daniel Collins
Daniel Hahler
Daniel Holth
Daniel Jost
Daniel Katz
Daniel Shaulov
Daniele Esposti
Daniele Nicolodi
Daniele Procida
Daniil Konovalenko
Danny Hermes
Danny McClanahan
Darren Kavanagh
Dav Clark
Dave Abrahams
Dave Jones
David Aguilar
David Black
David Bordeynik
David Caro
David D Lowe
David Evans
David Hewitt
David Linke
David Poggi
David Pursehouse
David Runge
David Tucker
David Wales
Davidovich
ddelange
Deepak Sharma
Deepyaman Datta
Denise Yu
dependabot[bot]
derwolfe
Desetude
Devesh Kumar Singh
Diego Caraballo
Diego Ramirez
DiegoCaraballo
Dimitri Merejkowsky
Dimitri Papadopoulos
Dirk Stolle
Dmitry Gladkov
Dmitry Volodin
Domen Kožar
Dominic Davis-Foster
Donald Stufft
Dongweiming
doron zarhi
Dos Moonen
Douglas Thor
DrFeathers
Dustin Ingram
Dwayne Bailey
Ed Morley
Edgar Ramírez
Edgar Ramírez Mondragón
Ee Durbin
Efflam Lemaillet
efflamlemaillet
Eitan Adler
ekristina
elainechan
Eli Schwartz
Elisha Hollander
Ellen Marie Dash
Emil Burzo
Emil Styrke
Emmanuel Arias
Endoh Takanao
enoch
Erdinc Mutlu
Eric Cousineau
Eric Gillingham
Eric Hanchrow
Eric Hopper
Erik M. Bray
Erik Rose
Erwin Janssen
Eugene Vereshchagin
everdimension
Federico
Felipe Peter
Felix Yan
fiber-space
Filip Kokosiński
Filipe Laíns
Finn Womack
finnagin
Flavio Amurrio
Florian Briand
Florian Rathgeber
Francesco
Francesco Montesano
Frost Ming
Gabriel Curio
Gabriel de Perthuis
Garry Polley
gavin
gdanielson
Geoffrey Sneddon
George Song
Georgi Valkov
Georgy Pchelkin
ghost
Giftlin Rajaiah
gizmoguy1
gkdoc
Godefroid Chapelle
Gopinath M
GOTO Hayato
gousaiyang
gpiks
Greg Roodt
Greg Ward
Guilherme Espada
Guillaume Seguin
gutsytechster
Guy Rozendorn
Guy Tuval
gzpan123
Hanjun Kim
Hari Charan
Harsh Vardhan
harupy
Harutaka Kawamura
hauntsaninja
Henrich Hartzer
Henry Schreiner
Herbert Pfennig
Holly Stotelmyer
Honnix
Hsiaoming Yang
Hugo Lopes Tavares
Hugo van Kemenade
Hugues Bruant
Hynek Schlawack
Ian Bicking
Ian Cordasco
Ian Lee
Ian Stapleton Cordasco
Ian Wienand
Igor Kuzmitshov
Igor Sobreira
Ilan Schnell
Illia Volochii
Ilya Baryshev
Inada Naoki
Ionel Cristian Mărieș
Ionel Maries Cristian
Itamar Turner-Trauring
Ivan Pozdeev
J. Nick Koston
Jacob Kim
Jacob Walls
Jaime Sanz
jakirkham
Jakub Kuczys
Jakub Stasiak
Jakub Vysoky
Jakub Wilk
James Cleveland
James Curtin
James Firth
James Gerity
James Polley
Jan Pokorný
Jannis Leidel
Jarek Potiuk
jarondl
Jason Curtis
Jason R. Coombs
JasonMo
JasonMo1
Jay Graves
Jean Abou Samra
Jean-Christophe Fillion-Robin
Jeff Barber
Jeff Dairiki
Jeff Widman
Jelmer Vernooij
jenix21
Jeremy Stanley
Jeremy Zafran
Jesse Rittner
Jiashuo Li
Jim Fisher
Jim Garrison
Jiun Bae
Jivan Amara
Joe Bylund
Joe Michelini
John Paton
John T. Wodder II
John-Scott Atlakson
johnthagen
Jon Banafato
Jon Dufresne
Jon Parise
Jonas Nockert
Jonathan Herbert
Joonatan Partanen
Joost Molenaar
Jorge Niedbalski
Joseph Bylund
Joseph Long
Josh Bronson
Josh Hansen
Josh Schneier
Joshua
Juan Luis Cano Rodríguez
Juanjo Bazán
Judah Rand
Julian Berman
Julian Gethmann
Julien Demoor
Jussi Kukkonen
jwg4
Jyrki Pulliainen
Kai Chen
Kai Mueller
Kamal Bin Mustafa
kasium
kaustav haldar
keanemind
Keith Maxwell
Kelsey Hightower
Kenneth Belitzky
Kenneth Reitz
Kevin Burke
Kevin Carter
Kevin Frommelt
Kevin R Patterson
Kexuan Sun
Kit Randel
Klaas van Schelven
KOLANICH
kpinc
Krishna Oza
Kumar McMillan
Kurt McKee
Kyle Persohn
lakshmanaram
Laszlo Kiss-Kollar
Laurent Bristiel
Laurent LAPORTE
Laurie O
Laurie Opperman
layday
Leon Sasson
Lev Givon
Lincoln de Sousa
Lipis
lorddavidiii
Loren Carvalho
Lucas Cimon
Ludovic Gasc
Lukas Geiger
Lukas Juhrich
Luke Macken
Luo Jiebin
luojiebin
luz.paz
László Kiss Kollár
M00nL1ght
Marc Abramowitz
Marc Tamlyn
Marcus Smith
Mariatta
Mark Kohler
Mark Williams
Markus Hametner
Martey Dodoo
Martin Fischer
Martin Häcker
Martin Pavlasek
Masaki
Masklinn
Matej Stuchlik
Mathew Jennings
Mathieu Bridon
Mathieu Kniewallner
Matt Bacchi
Matt Good
Matt Maker
Matt Robenolt
matthew
Matthew Einhorn
Matthew Feickert
Matthew Gilliard
Matthew Iversen
Matthew Treinish
Matthew Trumbell
Matthew Willson
Matthias Bussonnier
mattip
Maurits van Rees
Max W Chase
Maxim Kurnikov
Maxime Rouyrre
mayeut
mbaluna
mdebi
memoselyk
meowmeowcat
Michael
Michael Aquilina
Michael E. Karpeles
Michael Klich
Michael Mintz
Michael Williamson
michaelpacer
Michał Górny
Mickaël Schoentgen
Miguel Araujo Perez
Mihir Singh
Mike
Mike Hendricks
Min RK
MinRK
Miro Hrončok
Monica Baluna
montefra
Monty Taylor
Muha Ajjan‮
Nadav Wexler
Nahuel Ambrosini
Nate Coraor
Nate Prewitt
Nathan Houghton
Nathaniel J. Smith
Nehal J Wani
Neil Botelho
Nguyễn Gia Phong
Nicholas Serra
Nick Coghlan
Nick Stenning
Nick Timkovich
Nicolas Bock
Nicole Harris
Nikhil Benesch
Nikhil Ladha
Nikita Chepanov
Nikolay Korolev
Nipunn Koorapati
Nitesh Sharma
Niyas Sait
Noah
Noah Gorny
Nowell Strite
NtaleGrey
nvdv
OBITORASU
Ofek Lev
ofrinevo
Oliver Freund
Oliver Jeeves
Oliver Mannion
Oliver Tonnhofer
Olivier Girardot
Olivier Grisel
Ollie Rutherfurd
OMOTO Kenji
Omry Yadan
onlinejudge95
Oren Held
Oscar Benjamin
Oz N Tiram
Pachwenko
Patrick Dubroy
Patrick Jenkins
Patrick Lawson
patricktokeeffe
Patrik Kopkan
Paul Ganssle
Paul Kehrer
Paul Moore
Paul Nasrat
Paul Oswald
Paul van der Linden
Paulus Schoutsen
Pavel Safronov
Pavithra Eswaramoorthy
Pawel Jasinski
Paweł Szramowski
Pekka Klärck
Peter Gessler
Peter Lisák
Peter Waller
petr-tik
Phaneendra Chiruvella
Phil Elson
Phil Freo
Phil Pennock
Phil Whelan
Philip Jägenstedt
Philip Molloy
Philippe Ombredanne
Pi Delport
Pierre-Yves Rofes
Pieter Degroote
pip
Prabakaran Kumaresshan
Prabhjyotsing Surjit Singh Sodhi
Prabhu Marappan
Pradyun Gedam
Prashant Sharma
Pratik Mallya
pre-commit-ci[bot]
Preet Thakkar
Preston Holmes
Przemek Wrzos
Pulkit Goyal
q0w
Qiangning Hong
Qiming Xu
Quentin Lee
Quentin Pradet
R. David Murray
Rafael Caricio
Ralf Schmitt
Razzi Abuissa
rdb
Reece Dunham
Remi Rampin
Rene Dudfield
Riccardo Magliocchetti
Riccardo Schirone
Richard Jones
Richard Si
Ricky Ng-Adam
Rishi
RobberPhex
Robert Collins
Robert McGibbon
Robert Pollak
Robert T. McGibbon
robin elisha robinson
Roey Berman
Rohan Jain
Roman Bogorodskiy
Roman Donchenko
Romuald Brunet
ronaudinho
Ronny Pfannschmidt
Rory McCann
Ross Brattain
Roy Wellington Ⅳ
Ruairidh MacLeod
Russell Keith-Magee
Ryan Shepherd
Ryan Wooden
ryneeverett
Sachi King
Salvatore Rinchiera
sandeepkiran-js
Sander Van Balen
Savio Jomton
schlamar
Scott Kitterman
Sean
seanj
Sebastian Jordan
Sebastian Schaetz
Segev Finer
SeongSoo Cho
Sergey Vasilyev
Seth Michael Larson
Seth Woodworth
Shahar Epstein
Shantanu
shireenrao
Shivansh-007
Shlomi Fish
Shovan Maity
Simeon Visser
Simon Cross
Simon Pichugin
sinoroc
sinscary
snook92
socketubs
Sorin Sbarnea
Srinivas Nyayapati
Stavros Korokithakis
Stefan Scherfke
Stefano Rivera
Stephan Erb
Stephen Rosen
stepshal
Steve (Gadget) Barnes
Steve Barnes
Steve Dower
Steve Kowalik
Steven Myint
Steven Silvester
stonebig
studioj
Stéphane Bidoul
Stéphane Bidoul (ACSONE)
Stéphane Klein
Sumana Harihareswara
Surbhi Sharma
Sviatoslav Sydorenko
Swat009
Sylvain
Takayuki SHIMIZUKAWA
Taneli Hukkinen
tbeswick
Thiago
Thijs Triemstra
Thomas Fenzl
Thomas Grainger
Thomas Guettler
Thomas Johansson
Thomas Kluyver
Thomas Smith
Thomas VINCENT
Tim D. Smith
Tim Gates
Tim Harder
Tim Heap
tim smith
tinruufu
Tobias Hermann
Tom Forbes
Tom Freudenheim
Tom V
Tomas Hrnciar
Tomas Orsava
Tomer Chachamu
Tommi Enenkel | AnB
Tomáš Hrnčiar
Tony Beswick
Tony Narlock
Tony Zhaocheng Tan
TonyBeswick
toonarmycaptain
Toshio Kuratomi
toxinu
Travis Swicegood
Tushar Sadhwani
Tzu-ping Chung
Valentin Haenel
Victor Stinner
victorvpaulo
Vikram - Google
Viktor Szépe
Ville Skyttä
Vinay Sajip
Vincent Philippon
Vinicyus Macedo
Vipul Kumar
Vitaly Babiy
Vladimir Fokow
Vladimir Rutsky
W. Trevor King
Wil Tan
Wilfred Hughes
William Edwards
William ML Leslie
William T Olson
William Woodruff
Wilson Mo
wim glenn
Winson Luk
Wolfgang Maier
Wu Zhenyu
XAMES3
Xavier Fernandez
xoviat
xtreak
YAMAMOTO Takashi
Yen Chi Hsuan
Yeray Diaz Diaz
Yoval P
Yu Jian
Yuan Jing Vincent Yan
Yusuke Hayashi
Zearin
Zhiping Deng
ziebam
Zvezdan Petkovic
Łukasz Langa
Роман Донченко
Семён Марьясин
‮rekcäH nitraM‮

1
.venv/lib/python3.12/site-packages/pip-24.0.dist-info/INSTALLER Normal file
View File

@@ -0,0 +1 @@
pip

20
.venv/lib/python3.12/site-packages/pip-24.0.dist-info/LICENSE.txt Normal file
View File

@@ -0,0 +1,20 @@
Copyright (c) 2008-present The pip developers (see AUTHORS.txt file)
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

88
.venv/lib/python3.12/site-packages/pip-24.0.dist-info/METADATA Normal file
View File

@@ -0,0 +1,88 @@
Metadata-Version: 2.1
Name: pip
Version: 24.0
Summary: The PyPA recommended tool for installing Python packages.
Author-email: The pip developers <distutils-sig@python.org>
License: MIT
Project-URL: Homepage, https://pip.pypa.io/
Project-URL: Documentation, https://pip.pypa.io
Project-URL: Source, https://github.com/pypa/pip
Project-URL: Changelog, https://pip.pypa.io/en/stable/news/
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Software Development :: Build Tools
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.7
Description-Content-Type: text/x-rst
License-File: LICENSE.txt
License-File: AUTHORS.txt
pip - The Python Package Installer
==================================
.. image:: https://img.shields.io/pypi/v/pip.svg
:target: https://pypi.org/project/pip/
:alt: PyPI
.. image:: https://img.shields.io/pypi/pyversions/pip
:target: https://pypi.org/project/pip
:alt: PyPI - Python Version
.. image:: https://readthedocs.org/projects/pip/badge/?version=latest
:target: https://pip.pypa.io/en/latest
:alt: Documentation
pip is the `package installer`_ for Python. You can use pip to install packages from the `Python Package Index`_ and other indexes.
Please take a look at our documentation for how to install and use pip:
* `Installation`_
* `Usage`_
We release updates regularly, with a new version every 3 months. Find more details in our documentation:
* `Release notes`_
* `Release process`_
If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms:
* `Issue tracking`_
* `Discourse channel`_
* `User IRC`_
If you want to get involved head over to GitHub to get the source code, look at our development documentation and feel free to jump on the developer mailing lists and chat rooms:
* `GitHub page`_
* `Development documentation`_
* `Development IRC`_
Code of Conduct
---------------
Everyone interacting in the pip project's codebases, issue trackers, chat
rooms, and mailing lists is expected to follow the `PSF Code of Conduct`_.
.. _package installer: https://packaging.python.org/guides/tool-recommendations/
.. _Python Package Index: https://pypi.org
.. _Installation: https://pip.pypa.io/en/stable/installation/
.. _Usage: https://pip.pypa.io/en/stable/
.. _Release notes: https://pip.pypa.io/en/stable/news.html
.. _Release process: https://pip.pypa.io/en/latest/development/release-process/
.. _GitHub page: https://github.com/pypa/pip
.. _Development documentation: https://pip.pypa.io/en/latest/development
.. _Issue tracking: https://github.com/pypa/pip/issues
.. _Discourse channel: https://discuss.python.org/c/packaging
.. _User IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa
.. _Development IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa-dev
.. _PSF Code of Conduct: https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md

1005
.venv/lib/python3.12/site-packages/pip-24.0.dist-info/RECORD Normal file
View File

File diff suppressed because it is too large Load Diff

0
.venv/lib/python3.12/site-packages/pip-24.0.dist-info/REQUESTED Normal file
View File

5
.venv/lib/python3.12/site-packages/pip-24.0.dist-info/WHEEL Normal file
View File

@@ -0,0 +1,5 @@
Wheel-Version: 1.0
Generator: bdist_wheel (0.42.0)
Root-Is-Purelib: true
Tag: py3-none-any

Some files were not shown because too many files have changed in this diff Show More