Leon/docutranslate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

增加赞赏渠道

Browse Source
This commit is contained in:
xunbu
2025-09-07 14:39:52 +08:00
parent 3385863244
commit b493f7d4ae
4 changed files with 185 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files

151
README.md
View File

@@ -20,18 +20,26 @@
A lightweight local file translation tool based on a large language model.
</p>
- ✅ **Supports Multiple Formats**: Can translate various files such as `pdf`, `docx`, `xlsx`, `md`, `txt`, `json`, `epub`, `srt`, and more.
- ✅ **Supports Multiple Formats**: Can translate various files such as `pdf`, `docx`, `xlsx`, `md`, `txt`, `json`,
`epub`, `srt`, and more.
- ✅ **Automatic Glossary Generation**: Supports automatic generation of glossaries to ensure term alignment.
- ✅ **PDF Table, Formula, and Code Recognition**: With the `docling` and `mineru` PDF parsing engines, it can recognize and translate tables, formulas, and code frequently found in academic papers.
- ✅ **JSON Translation**: Supports specifying the values to be translated in JSON via JSON paths (using `jsonpath-ng` syntax).
- ✅ **Word/Excel Format-Preserving Translation**: Supports translating `docx` and `xlsx` files (currently not `doc` or `xls` files) while preserving the original formatting.
- ✅ **Multi-AI Platform Support**: Supports most AI platforms, enabling high-performance, concurrent AI translation with custom prompts.
- ✅ **Asynchronous Support**: Designed for high-performance scenarios, it offers complete asynchronous support, providing service interfaces for parallel multitasking.
- ✅ **PDF Table, Formula, and Code Recognition**: With the `docling` and `mineru` PDF parsing engines, it can recognize
and translate tables, formulas, and code frequently found in academic papers.
- ✅ **JSON Translation**: Supports specifying the values to be translated in JSON via JSON paths (using `jsonpath-ng`
syntax).
- ✅ **Word/Excel Format-Preserving Translation**: Supports translating `docx` and `xlsx` files (currently not `doc` or
`xls` files) while preserving the original formatting.
- ✅ **Multi-AI Platform Support**: Supports most AI platforms, enabling high-performance, concurrent AI translation with
custom prompts.
- ✅ **Asynchronous Support**: Designed for high-performance scenarios, it offers complete asynchronous support,
providing service interfaces for parallel multitasking.
- ✅ **LAN and Multi-user Support**: Supports simultaneous use by multiple users on a local area network.
- ✅ **Interactive Web Interface**: Provides an out-of-the-box Web UI and RESTful API for easy integration and use.
- ✅ **Small-Footprint, Multi-Platform "Lazy" Packages**: Windows and Mac "lazy" packages under 40MB (for versions not using `docling` for local PDF parsing).
- ✅ **Small-Footprint, Multi-Platform "Lazy" Packages**: Windows and Mac "lazy" packages under 40MB (for versions not
using `docling` for local PDF parsing).
> When translating `pdf` files, they are first converted to Markdown, which will **lose** the original layout. Users with layout requirements should take note.
> When translating `pdf` files, they are first converted to Markdown, which will **lose** the original layout. Users
> with layout requirements should take note.
> QQ Discussion Group: 1047781902
@@ -45,10 +53,14 @@
## All-in-One Packages
For users who want to get started quickly, we provide all-in-one packages on [GitHub Releases](https://github.com/xunbu/docutranslate/releases). Simply download, unzip, and enter your AI platform API-Key to start using.
For users who want to get started quickly, we provide all-in-one packages
on [GitHub Releases](https://github.com/xunbu/docutranslate/releases). Simply download, unzip, and enter your AI
platform API-Key to start using.
- **DocuTranslate**: Standard version, uses the online `minerU` engine to parse PDF documents. Choose this version if you don't need local PDF parsing (recommended).
- **DocuTranslate_full**: Full version, includes the built-in `docling` local PDF parsing engine. Choose this version if you need local PDF parsing.
- **DocuTranslate**: Standard version, uses the online `minerU` engine to parse PDF documents. Choose this version if
you don't need local PDF parsing (recommended).
- **DocuTranslate_full**: Full version, includes the built-in `docling` local PDF parsing engine. Choose this version if
you need local PDF parsing.
## Installation
@@ -88,31 +100,36 @@ uv sync
## Core Concept: Workflow
The core of the new DocuTranslate is the **Workflow**. Each workflow is a complete, end-to-end translation pipeline designed specifically for a particular file type. You no longer interact with a monolithic class; instead, you select and configure a suitable workflow based on your file type.
The core of the new DocuTranslate is the **Workflow**. Each workflow is a complete, end-to-end translation pipeline
designed specifically for a particular file type. You no longer interact with a monolithic class; instead, you select
and configure a suitable workflow based on your file type.
**The basic usage process is as follows:**
1. **Select a Workflow**: Choose a workflow based on your input file type (e.g., PDF/Word or TXT), such as `MarkdownBasedWorkflow` or `TXTWorkflow`.
2. **Build the Configuration**: Create the corresponding configuration object for the selected workflow (e.g., `MarkdownBasedWorkflowConfig`). This configuration object contains all the necessary sub-configurations, such as:
1. **Select a Workflow**: Choose a workflow based on your input file type (e.g., PDF/Word or TXT), such as
`MarkdownBasedWorkflow` or `TXTWorkflow`.
2. **Build the Configuration**: Create the corresponding configuration object for the selected workflow (e.g.,
`MarkdownBasedWorkflowConfig`). This configuration object contains all the necessary sub-configurations, such as:
* **Converter Config**: Defines how to convert the original file (e.g., PDF) to Markdown.
* **Translator Config**: Defines which LLM, API-Key, target language, etc., to use.
* **Exporter Config**: Defines specific options for the output format (e.g., HTML).
3. **Instantiate the Workflow**: Create a workflow instance using the configuration object.
4. **Execute the Translation**: Call the workflow's `.read_*()` and `.translate()` / `.translate_async()` methods.
5. **Export/Save the Result**: Call the `.export_to_*()` or `.save_as_*()` methods to get or save the translation result.
5. **Export/Save the Result**: Call the `.export_to_*()` or `.save_as_*()` methods to get or save the translation
result.
## Available Workflows
| Workflow | Use Case | Input Formats | Output Formats | Core Config Class |
|:---|:---|:---|:---|:---|
| Workflow | Use Case | Input Formats | Output Formats | Core Config Class |
|:----------------------------|:----------------------------------------------------------------------------------------------------------------------|:---------------------------------------------|:-----------------------|:------------------------------|
| **`MarkdownBasedWorkflow`** | Processes rich text documents like PDF, Word, images, etc. The process is: `File -> Markdown -> Translate -> Export`. | `.pdf`, `.docx`, `.md`, `.png`, `.jpg`, etc. | `.md`, `.zip`, `.html` | `MarkdownBasedWorkflowConfig` |
| **`TXTWorkflow`** | Processes plain text documents. The process is: `txt -> Translate -> Export`. | `.txt` and other plain text formats | `.txt`, `.html` | `TXTWorkflowConfig` |
| **`JsonWorkflow`** | Processes JSON files. The process is: `json -> Translate -> Export`. | `.json` | `.json`, `.html` | `JsonWorkflowConfig` |
| **`DocxWorkflow`** | Processes docx files. The process is: `docx -> Translate -> Export`. | `.docx` | `.docx`, `.html` | `docxWorkflowConfig` |
| **`XlsxWorkflow`** | Processes xlsx files. The process is: `xlsx -> Translate -> Export`. | `.xlsx`, `.csv` | `.xlsx`, `.html` | `XlsxWorkflowConfig` |
| **`SrtWorkflow`** | Processes srt files. The process is: `srt -> Translate -> Export`. | `.srt` | `.srt`, `.html` | `SrtWorkflowConfig` |
| **`EpubWorkflow`** | Processes epub files. The process is: `epub -> Translate -> Export`. | `.epub` | `.epub`, `.html` | `EpubWorkflowConfig` |
| **`HtmlWorkflow`** | Processes html files. The process is: `html -> Translate -> Export`. | `.html`, `.htm` | `.html` | `HtmlWorkflowConfig` |
| **`TXTWorkflow`** | Processes plain text documents. The process is: `txt -> Translate -> Export`. | `.txt` and other plain text formats | `.txt`, `.html` | `TXTWorkflowConfig` |
| **`JsonWorkflow`** | Processes JSON files. The process is: `json -> Translate -> Export`. | `.json` | `.json`, `.html` | `JsonWorkflowConfig` |
| **`DocxWorkflow`** | Processes docx files. The process is: `docx -> Translate -> Export`. | `.docx` | `.docx`, `.html` | `docxWorkflowConfig` |
| **`XlsxWorkflow`** | Processes xlsx files. The process is: `xlsx -> Translate -> Export`. | `.xlsx`, `.csv` | `.xlsx`, `.html` | `XlsxWorkflowConfig` |
| **`SrtWorkflow`** | Processes srt files. The process is: `srt -> Translate -> Export`. | `.srt` | `.srt`, `.html` | `SrtWorkflowConfig` |
| **`EpubWorkflow`** | Processes epub files. The process is: `epub -> Translate -> Export`. | `.epub` | `.epub`, `.html` | `EpubWorkflowConfig` |
| **`HtmlWorkflow`** | Processes html files. The process is: `html -> Translate -> Export`. | `.html`, `.htm` | `.html` | `HtmlWorkflowConfig` |
> In the interactive interface, you can export to PDF format.
@@ -134,14 +151,16 @@ export DOCUTRANSLATE_PORT=8011
docutranslate -i
```
- **Interactive Interface**: After starting the service, please visit `http://127.0.0.1:8010` (or your specified port) in your browser.
- **Interactive Interface**: After starting the service, please visit `http://127.0.0.1:8010` (or your specified port)
in your browser.
- **API Documentation**: The complete API documentation (Swagger UI) is available at `http://127.0.0.1:8010/docs`.
## Usage
### Example 1: Translating a PDF File (using `MarkdownBasedWorkflow`)
This is the most common use case. We will use the `minerU` engine to convert the PDF to Markdown, and then use an LLM for translation. Here is an example using the asynchronous approach.
This is the most common use case. We will use the `minerU` engine to convert the PDF to Markdown, and then use an LLM
for translation. Here is an example using the asynchronous approach.
```python
import asyncio
@@ -207,7 +226,8 @@ if __name__ == "__main__":
### Example 2: Translating a TXT File (using `TXTWorkflow`)
For plain text files, the process is simpler as it doesn't require a document parsing (conversion) step. Here is an example using the asynchronous approach.
For plain text files, the process is simpler as it doesn't require a document parsing (conversion) step. Here is an
example using the asynchronous approach.
```python
import asyncio
@@ -254,7 +274,9 @@ if __name__ == "__main__":
### Example 3: Translating a JSON File (using `JsonWorkflow`)
Here is an example using the asynchronous approach. The `json_paths` item in `JsonTranslatorConfig` needs to specify the JSON paths to be translated (satisfying the `jsonpath-ng` syntax). Only values matching the JSON paths will be translated.
Here is an example using the asynchronous approach. The `json_paths` item in `JsonTranslatorConfig` needs to specify the
JSON paths to be translated (satisfying the `jsonpath-ng` syntax). Only values matching the JSON paths will be
translated.
```python
import asyncio
@@ -405,24 +427,26 @@ if __name__ == "__main__":
### 1. Get a Large Language Model API Key
The translation functionality relies on large language models. You need to obtain a `base_url`, `api_key`, and `model_id` from the respective AI platform.
The translation functionality relies on large language models. You need to obtain a `base_url`, `api_key`, and
`model_id` from the respective AI platform.
> Recommended models: Volcengine's `doubao-seed-1-6-250615`, `doubao-seed-1-6-flash-250715`, Zhipu's `glm-4-flash`, Alibaba Cloud's `qwen-plus`, `qwen-turbo`, Deepseek's `deepseek-chat`, etc.
> Recommended models: Volcengine's `doubao-seed-1-6-250615`, `doubao-seed-1-6-flash-250715`, Zhipu's `glm-4-flash`,
> Alibaba Cloud's `qwen-plus`, `qwen-turbo`, Deepseek's `deepseek-chat`, etc.
| Platform Name | Get API Key | baseurl |
|---|---|---|
| ollama | | http://127.0.0.1:11434/v1 |
| lm studio | | http://127.0.0.1:1234/v1 |
| openrouter | [Click to get](https://openrouter.ai/settings/keys) | https://openrouter.ai/api/v1 |
| openai | [Click to get](https://platform.openai.com/api-keys) | https://api.openai.com/v1/ |
| gemini | [Click to get](https://aistudio.google.com/u/0/apikey) | https://generativelanguage.googleapis.com/v1beta/openai/ |
| deepseek | [Click to get](https://platform.deepseek.com/api_keys) | https://api.deepseek.com/v1 |
| Zhipu AI | [Click to get](https://open.bigmodel.cn/usercenter/apikeys) | https://open.bigmodel.cn/api/paas/v4 |
| Tencent Hunyuan | [Click to get](https://console.cloud.tencent.com/hunyuan/api-key) | https://api.hunyuan.cloud.tencent.com/v1 |
| Alibaba Cloud Bailian | [Click to get](https://bailian.console.aliyun.com/?tab=model#/api-key) | https://dashscope.aliyuncs.com/compatible-mode/v1 |
| Volcengine | [Click to get](https://console.volcengine.com/ark/region:ark+cn-beijing/apiKey?apikey=%7B%7D) | https://ark.cn-beijing.volces.com/api/v3 |
| SiliconFlow | [Click to get](https://cloud.siliconflow.cn/account/ak) | https://api.siliconflow.cn/v1 |
| DMXAPI | [Click to get](https://www.dmxapi.cn/token) | https://www.dmxapi.cn/v1 |
| Platform Name | Get API Key | baseurl |
|-----------------------|-----------------------------------------------------------------------------------------------|----------------------------------------------------------|
| ollama | | http://127.0.0.1:11434/v1 |
| lm studio | | http://127.0.0.1:1234/v1 |
| openrouter | [Click to get](https://openrouter.ai/settings/keys) | https://openrouter.ai/api/v1 |
| openai | [Click to get](https://platform.openai.com/api-keys) | https://api.openai.com/v1/ |
| gemini | [Click to get](https://aistudio.google.com/u/0/apikey) | https://generativelanguage.googleapis.com/v1beta/openai/ |
| deepseek | [Click to get](https://platform.deepseek.com/api_keys) | https://api.deepseek.com/v1 |
| Zhipu AI | [Click to get](https://open.bigmodel.cn/usercenter/apikeys) | https://open.bigmodel.cn/api/paas/v4 |
| Tencent Hunyuan | [Click to get](https://console.cloud.tencent.com/hunyuan/api-key) | https://api.hunyuan.cloud.tencent.com/v1 |
| Alibaba Cloud Bailian | [Click to get](https://bailian.console.aliyun.com/?tab=model#/api-key) | https://dashscope.aliyuncs.com/compatible-mode/v1 |
| Volcengine | [Click to get](https://console.volcengine.com/ark/region:ark+cn-beijing/apiKey?apikey=%7B%7D) | https://ark.cn-beijing.volces.com/api/v3 |
| SiliconFlow | [Click to get](https://cloud.siliconflow.cn/account/ak) | https://api.siliconflow.cn/v1 |
| DMXAPI | [Click to get](https://www.dmxapi.cn/token) | https://www.dmxapi.cn/v1 |
### 2. PDF Parsing Engine (ignore if not translating PDFs)
@@ -437,18 +461,23 @@ If you choose `mineru` as the document parsing engine (`convert_engine="mineru"`
### 2.2. docling Engine Configuration (local PDF parsing)
If you choose `docling` as the document parsing engine (`convert_engine="docling"`), it will download the required models from Hugging Face on first use.
If you choose `docling` as the document parsing engine (`convert_engine="docling"`), it will download the required
models from Hugging Face on first use.
> A better option is to download `docling_artifact.zip` from [GitHub releases](https://github.com/xunbu/docutranslate/releases) and unzip it to your working directory.
> A better option is to download `docling_artifact.zip`
> from [GitHub releases](https://github.com/xunbu/docutranslate/releases) and unzip it to your working directory.
**Solution for network issues when downloading `docling` models:**
1. **Set a Hugging Face mirror (recommended)**:
* **Method A (environment variable)**: Set the system environment variable `HF_ENDPOINT` and restart your IDE or terminal.
* **Method A (environment variable)**: Set the system environment variable `HF_ENDPOINT` and restart your IDE or
terminal.
```
HF_ENDPOINT=https://hf-mirror.com
```
* **Method B (set in code)**: Add the following code at the beginning of your Python script.
```python
import os
@@ -456,9 +485,11 @@ os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
```
2. **Offline use (download the model package in advance)**:
* Download `docling_artifact.zip` from [GitHub Releases](https://github.com/xunbu/docutranslate/releases).
* Unzip it to your project directory.
* Specify the model path in the configuration (if the model is not in the same directory as the script):
```python
from docutranslate.converter.x2md.converter_docling import ConverterDoclingConfig
@@ -478,19 +509,25 @@ A: Use the `-p` parameter to specify a new port, or set the `DOCUTRANSLATE_PORT`
A: Yes. Please use the `mineru` parsing engine, which has powerful OCR capabilities.
**Q: Why is the first PDF translation so slow?**
A: If you are using the `docling` engine, it needs to download models from Hugging Face on its first run. Please refer to the "Network Issues Solution" above to speed up this process.
A: If you are using the `docling` engine, it needs to download models from Hugging Face on its first run. Please refer
to the "Network Issues Solution" above to speed up this process.
**Q: How to use it in an intranet (offline) environment?**
A: It is entirely possible. You need to meet the following conditions:
1. **Local LLM**: Use tools like [Ollama](https://ollama.com/) or [LM Studio](https://lmstudio.ai/) to deploy a language model locally, and fill in the `base_url` of the local model in `TranslatorConfig`.
2. **Local PDF parsing engine** (only needed for parsing PDFs): Use the `docling` engine and follow the "Offline use" instructions above to download the model package in advance.
1. **Local LLM**: Use tools like [Ollama](https://ollama.com/) or [LM Studio](https://lmstudio.ai/) to deploy a language
model locally, and fill in the `base_url` of the local model in `TranslatorConfig`.
2. **Local PDF parsing engine** (only needed for parsing PDFs): Use the `docling` engine and follow the "Offline use"
instructions above to download the model package in advance.
**Q: How does the PDF parsing cache mechanism work?**
A: `MarkdownBasedWorkflow` automatically caches the results of document parsing (file to Markdown conversion) to avoid repeated parsing that consumes time and resources. The cache is stored in memory by default and records the last 10 parses. You can modify the cache size using the `DOCUTRANSLATE_CACHE_NUM` environment variable.
A: `MarkdownBasedWorkflow` automatically caches the results of document parsing (file to Markdown conversion) to avoid
repeated parsing that consumes time and resources. The cache is stored in memory by default and records the last 10
parses. You can modify the cache size using the `DOCUTRANSLATE_CACHE_NUM` environment variable.
**Q: How to make the software go through a proxy?**
A: The software does not use a proxy by default. You can enable it by setting the environment variable `DOCUTRANSLATE_PROXY_ENABLED` to `true`.
A: The software does not use a proxy by default. You can enable it by setting the environment variable
`DOCUTRANSLATE_PROXY_ENABLED` to `true`.
## Star History
@@ -500,4 +537,12 @@ A: The software does not use a proxy by default. You can enable it by setting th
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=xunbu/docutranslate&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=xunbu/docutranslate&type=Date" />
</picture>
</a>
</a>
## Support the Author
Welcome to support the author. Please kindly mention the reason for your appreciation in the notes. ❤
<p align="center">
<img src="./images/赞赏码.jpg" alt="赞赏码" style="width: 50vw">
</p>