Hi @wannaphong, you did not specify those changes before. I made the changes as you requested. Feel free to ask if you need further changes.

Can you add those functions to sent_tokenize?

def indices_words(words):
    indices = []
    start_index = 0

    for word in words:
        if len(word)>1:
            _temp= len(word)-1
        else:
            _temp=1
        indices.append((start_index, start_index + _temp))
        start_index += len(word)

    return indices
list_word = ['ผม', 'กินข้าว', ' ', 'เธอ', 'เล่น', 'เกม']
index=indices_words(list_word)  # [(0, 1), (2, 8), (9, 10), (10, 12), (13, 16), (17, 19)]
list_sent=sent_tokenize(list_word)  # ['ผมกินข้าว ', 'เธอเล่นเกม']

import copy
def map_indices_to_words(index_list, sentences):
    result = []
    c=copy.copy(index_list)
    n_sum=0
    
    for sentence in sentences:
        words = sentence
        sentence_result = []
        n=0
        
        for start, end in c:
            if start > n_sum+len(words)-1:
                break
            else:
                word = sentence[start-n_sum:end+1-n_sum]
                sentence_result.append(word)
                n+=1

        result.append(sentence_result)
        n_sum+=len(words)
        for _ in range(n):
            del c[0]
        
    
    return result
list_sent_word = map_indices_to_words(index,list_sent)  #  [['ผม', 'กินข้าว', ' '], ['เธอ', 'เล่น', 'เกม']]

@wannaphong , I'm a bit confused.

1. where do you want to implement 'map_indices_to_words' and 'indices_words' functions?
2. last changes for 'sent_tokenize' function are fine?

3. list_word = ['ผม', 'กินข้าว', ' ', 'เธอ', 'เล่น', 'เกม']
   index=indices_words(list_word) # [(0, 1), (2, 8), (9, 10), (10, 12), (13, 16), (17, 19)]
   list_sent=sent_tokenize(list_word) # ['ผมกินข้าว ', 'เธอเล่นเกม']

• current sent_tokenize return that output (['ผมกินข้าว ', 'เธอเล่นเกม']) if keep_whitespace is false.

@wannaphong , I'm a bit confused.

1. where do you want to implement 'map_indices_to_words' and 'indices_words' functions?

2. last changes for 'sent_tokenize' function are fine?

3. > list_word = ['ผม', 'กินข้าว', ' ', 'เธอ', 'เล่น', 'เกม']
   > index=indices_words(list_word)  # [(0, 1), (2, 8), (9, 10), (10, 12), (13, 16), (17, 19)]
   > list_sent=sent_tokenize(list_word)  # ['ผมกินข้าว ', 'เธอเล่นเกม']


* current sent_tokenize return that output (['ผมกินข้าว ', 'เธอเล่นเกม']) if keep_whitespace is false.

Yes, it can still keep list of words. Every sentence tokenizer engine have different word tokenizer inside sentence tokenizer but the engine can output as list of sentence only, so we can keep list of words to restore list of words in list of sentences.

Example: I used deepcut tokenizer for my text and used crfcut that use newmm inside crfcut engine. The output is list of sentences. If I use map_indices_to_words, it can output as list of words (from deepcut) in list of sentences.

Sorry, but I do not have any idea about the tokenization engine and how it works. Could you please provide more details? Then maybe I could help you with this.

Sorry, but I do not have any idea about the tokenization engine and how it works. Could you please provide more details? Then maybe I could help you with this.

from https://github.com/PyThaiNLP/pythainlp/blob/dev/pythainlp/tokenize/core.py#L325-L507

Here are the sentence tokenization engines:

• crfcut - (default) split by CRF trained on TED dataset
• thaisum - The implementation of sentence segmenter from Nakhun Chumpolsathien, 2020
• tltk - split by TLTK <https://pypi.org/project/tltk/>_.,
• wtp - split by wtpsplita <https://github.com/bminixhofer/wtpsplit>_.
• whitespace+newline - split by whitespace and newline.
• whitespace - split by whitespace, specifically with regex pattern r" +"

crfcut, thaisum, tltk, and wtp has own our word tokenizer inside the engine, so i think it should preprocessing by store indices word index and use the indices to tokenize word in list of sentences.

List of words ( ['ผม', 'กินข้าว', ' ', 'เธอ', 'เล่น', 'เกม']) -> Text ("ผมกินข้าว เธอเล่นเกม" and store indices)-> sent tokenizer -> List of sentences (['ผมกินข้าว ', 'เธอเล่นเกม']) -> List of sentences and list of word inside the list ([['ผม', 'กินข้าว', ' '], ['เธอ', 'เล่น', 'เกม']])

Hello @ayaan-qadri! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file pythainlp/tokenize/core.py:

Line 332:30: E226 missing whitespace around arithmetic operator
Line 352:29: E226 missing whitespace around arithmetic operator
Line 352:40: E226 missing whitespace around arithmetic operator
Line 355:38: E226 missing whitespace around arithmetic operator
Line 355:48: E226 missing whitespace around arithmetic operator
Line 355:50: E226 missing whitespace around arithmetic operator

Comment last updated at 2024-10-13 17:37:56 UTC

@wannaphong , Thanks for the explanation, Please check again, I have made mistake in past commit too sorry for that.

I found a bug.

The indices store whitespace but whitespace and whitespace+newline are tokenize by whitespace. Can you re-create word_indices in those engines?

word_indices = indices_words(text that are without whitespace in whitespace and whitespace+newline in whitespace+newline)

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@wannaphong , Check it now.

I found two bug in two case.

list_word=["ผม","กิน","ข้าว"," ","\n","เธอ","เล่น","เกม"]
sent_tokenize(list_word)

Results: [['ผม', 'กิน', 'ข้าว', ' \n', '\nเ', 'เธอ', 'เล่น', 'เกม']]

Expected results: [['ผม', 'กิน', 'ข้าว', ' ', '\n', 'เธอ', 'เล่น', 'เกม']]

list_word=["ผม","กิน","ข้าว"," ","\n","เธอ","เล่น","เกม"]
sent_tokenize(list_word,engine="whitespace")

Results: [['ผม', 'กิน', 'ข้าว'], ['\nเธ', 'อเล่', 'นเก']]

Expected results: [['ผม', 'กิน', 'ข้าว'], ['\n', 'เธอ', 'เล่น', 'เกม']]