1. The recognition of mathematical formulas in pictures has the following main application scenarios:
1. Electronic education : The online education platform can automatically identify mathematical formulas in courseware, exercises and test papers through formula recognition technology, realize the editing, calculation and understanding of formulas, and assist teaching and learning.
2. Processing of scientific research papers: By identifying the mathematical formulas in the papers, formula editing, typesetting and linking can be realized , the readability and operability of the papers can be improved, and readers can easily understand the content of the papers.
3. Mathematical formula search: By identifying mathematical formulas in a large number of documents and web pages, formula databases and indexes can be constructed to realize search and retrieval based on formula content. This has high application value for research in the fields of mathematics and engineering.
4. Visual question answering system: In the visual question answering system, recognizing the mathematical formula in the image can help the system understand the content of the image, and the formula information is also taken into account in the process of answering the question, making the answer more accurate and comprehensive.
5. Auxiliary calculation: By recognizing handwritten or printed mathematical formulas , combined with handwriting recognition and formula calculation technology, it can realize automatic calculation and solution of handwritten formulas, and assist people to complete related work or learning tasks efficiently.
6. Visual reasoning: In many visual reasoning and formula derivation tasks, recognizing mathematical formulas in images or videos is a key step in understanding the reasoning process and results. Only by accurately identifying the formulas in each step can the whole reasoning process be simulated and reproduced.
7. Others: There are also applications such as note organization, formula input method, and visual chemical equation recognition .
In short, as one of the most important tools in science and engineering, mathematical formulas have broad application prospects in their recognition and understanding. This also provides a wide range of practical scenarios and data support for mathematical formula recognition and related research.
Two, CeleryMath installation
2.1 Download address
After pasting the recognition result of version CeleryMath v0.1.4.1, you need to add "$" at the beginning and end: "$paste content $", and paste the latex output of version 0.1.2 to vscode and use it directly.
https://github.com/MODCT/CeleryMath/releases/download/v0.1.4.1/celeryMath-v0.1.4-Windows_X64.7z
2.2 Download celeryMah-weights-v0.1.0.zip
URL: https://github.com/MODCT/CeleryMath/releases/tag/v0.1.0
After decompression, celeryMathEncoder.onnx and celeryMathDecoder.onnx are obtained.
2.3 Download source code
git clone https://github.com/MODCT/CeleryMath.git
Copy the tokenizer.json in the source code together with celeryMathEncoder.onnx and celeryMathDecoder.onnx to celeryMath.dist/conf.
2.4 Modify conf/conf.json as follows:
{
"tokenizer_path": "conf/tokenizer.json",
"encoder_path": "conf/celeryMathEncoder.onnx",
"decoder_path": "conf/celeryMathDecoder.onnx",
"snip_hotkey": "Ctrl+Alt+S",
"search_method": "greedy",
"sampling": "nucleus",
"temperature": 0.2,
"beam_width": 5,
"device": "cpu",
"pad_token": 0,
"bos_token": 1,
"eos_token": 2,
"max_seq": 512,
"min_img_size": [
32,
32
],
"max_img_size": [
192,
896
]
}(optional) 2.5. Download the gpu version of onnxruntime
Version 0.1.4 added GPU support. (CUDA only)
To use the gpu, you need to download the onnxruntime gpu library from onnxruntime, and put
lib/onnxruntime_providers_cuda.dll 到 onnxruntime/capi/
https://github.com/microsoft/onnxruntime/releases
Finally, double-click celeryMath.exe to open the main interface.
Three, use
Example 1.
screenshot recognition
Copy to get:
\mathbf { j } = { \frac { \hbar } { 2 m i } } ( \psi ^ { * } \nabla \psi - \psi \nabla \psi ^ { * } )
Right-click the formula, copy latex to the clipboard
$Paste Content$: The first and last "$" need to be added manually
Example 2.
The End