Use data with limited resource

(Document WIP) This instruction talks about how you should use the data when having limited resource available (e.g. limited memory, limited storage space).

Intuition

Revision-based dataset could be extremely huge. For example, the basic Wikipedia Edit History dump is around 25TB when decompressed. So it is really unfriendly to decompress all warehouses at once and use them for training.

Last updated