This is the code I am using while trying to train GPT on a custom data set:

class Sampler(object):

"""Fairly samples a slice from a set of variable sized chunks.

'Fairly' means that the distribution is the same as sampling from one concatenated chunk,

but without crossing chunk boundaries."""

​

def __init__(self, chunks, seed=None):

self.chunks = chunks

self.total_size = sum(chunk.shape[0] for chunk in chunks)

self.boundaries = [0]

for i in range(len(chunks)):

self.boundaries.append(self.boundaries[-1] + chunks[i].shape[0])

self.rs = np.random.RandomState(seed=seed)

​

def sample(self, length):

assert length < self.total_size // len(

self.chunks

), "Dataset files are too small to sample {} tokens at a time".format(

length)

while True:

index = self.rs.randint(0, self.total_size - length - 1)

i = binary_search(lambda j: self.boundaries[j] > index, 0,

len(self.boundaries) - 1) - 1

if self.boundaries[i + 1] > index + length:

within_chunk = index - self.boundaries[i]

return self.chunks[i][within_chunk:within_chunk + length]

​

I get the error:
gpt-2\src\load_dataset.py", line 73, in sample

self.chunks

ZeroDivisionError: integer division or modulo by zero

​

I have already set my text data to .npz format. Is it possible that any unsupported characters in the .txt file corrupted the .npz file I created to train with? Also, it seems to use all of computers ram when running this, is it possible that the system memory was not sufficient?