How do I use my own dataset to calculate word weight?

[silear]

How do I use my own dataset to calculate word weight? Do I need to create a new term.freq? What is the difference between term.freq and huqie.txt? What is the style of term.freq?
如何使用我自己的数据集来计算词权，需要创建一个新的term.freq吗，term.freq与huqie.txt在用途上的区别是什么呢？term.freq的样式是什么样呢

[dosubot[bot]]

Hello @silear! I'm here to help you solve bugs, answer questions, and become a contributor while you wait for a human maintainer. Let's tackle your issue together! 🤖

To use your own dataset to calculate word weight, you can integrate it into the existing system by modifying the Dealer class. Here's how you can do it:

1. Prepare your custom dataset: Format your dataset similarly to term.freq, with terms and their frequencies separated by tabs.

2. Load your custom dataset: Modify the Dealer class to load your dataset using a method like load_custom_dataset.

3. Integrate the dataset: Use the loaded dataset in the weights method, modifying functions like freq or df to incorporate your custom data.

4. Initialize the custom dataset: In the Dealer class constructor, initialize your dataset using the load_custom_dataset method.

Regarding the files:

• term.freq: This file contains terms and their frequencies, formatted as term\tfrequency. It's used to calculate document frequency values for terms ^[1].

• huqie.txt: This file stores custom terms or keywords to improve tokenizer accuracy. Each term is formatted as TERM 1024 nr. After modifying it, delete the huqie.txt.trie file to apply changes ^[2].

The style of term.freq is:

term1\t10
term2\t20

Each line contains a term followed by a tab character and a frequency number ^[3].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[silear]

What are the files in rag/res used for?@dosu

[silear]

What words are put in term.freq？ what words are put in huqie.txt? Please explain in detail.@dosu

[dosubot[bot]]

The files in the rag/res directory serve specific purposes:

1. term.freq: This file contains tab-separated terms and their frequencies, which are used for term frequency analysis or indexing within RAGFlow ^[1].

2. synonym.json: This file stores synonyms, which can be utilized to enhance search capabilities or document understanding in RAGFlow ^[1].

Additionally, the deepdoc subdirectory within rag/res contains the updown_concat_xgb.model file, an XGBoost model used by RAGFlow's PDF parser for deep document understanding ^[1].

Regarding the words included in these files:

• term.freq: The file includes terms and their frequencies, but the specific criteria or process for selecting these words is not detailed in the available information ^[1].

• huqie.txt: This file serves as the tokenizer dictionary where custom terms can be added. The format for adding terms is TERM 1024 nr. After modifying this file, the huqie.txt.trie file must be deleted for the changes to take effect ^[2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[ZhenhangTung]

@silear Could you explain your question more? Any screenshot would also help a lot.
It would also be super helpful if you could share your use cases — that way we can better understand what you're trying to achieve and give more targeted suggestions 🙌