io

thutil.io

Functions:

• combine_text_files –

  Combine text files into a single file in a memory-efficient. Read and write in chunks to avoid loading large files into memory

• download_rawtext –

  Download raw text from a URL.

combine_text_files(files: list[str], output_file: str, chunk_size: int = 1024)

Combine text files into a single file in a memory-efficient. Read and write in chunks to avoid loading large files into memory

Parameters:

• files (list[str]) –

  List of file paths to combine.

• output_file (str) –

  Path to the output file.

• chunk_size (int, default: 1024 ) –

  Size of each chunk in KB to read/write. Defaults to 1024 KB.

download_rawtext(url: str, outfile: str = None) -> str

Download raw text from a URL.