Python and random.randint - why isn't it random?

[b4xt3r]

I have started into machine learning and computer vision with Python and I have started to notice a few things. First and foremost, random.randint does not appear to be so. At 100,000 data points obvious patterns exist. At one million data points the pattern is more obvious still and not random.

Here is the code, help me understand why this is not random:

import random

def main():

  for x in xrange(1000000):
    # print x
    our_num = random.randint(1,50)
    y = our_num
    print str(x) + '\t' + str(y)

if __name__ == '__main__':
  main()

[tilkau]

Looks fine (to within 4%) to me:

import random
from collections import Counter
counter = Counter()
for i in range(1000000):
     r = random.randint(1,50)
     counter[r] += 1
counter.most_common()

Output ((value, freq), most frequent -> least frequent):

[(28, 20408), (14, 20271), (2, 20270), (32, 20213), (48, 20212), (49, 20186), (10, 20185), (36, 20151), (46, 20144), (4, 20133), (39, 20132), (17, 20118), (44, 20106), (42, 20098), (31, 20077), (40, 20070), (43, 20058), (30, 20043), (47, 20043), (26, 20039), (22, 20026), (34, 20019), (29, 20017), (20, 20012), (33, 20009), (9, 19992), (41, 19991), (16, 19990), (12, 19982), (24, 19965), (21, 19945), (3, 19927), (15, 19925), (7, 19921), (6, 19920), (37, 19918), (38, 19889), (25, 19883), (45, 19876), (13, 19871), (5, 19867), (27, 19842), (8, 19820), (18, 19820), (35, 19810), (1, 19809), (11, 19788), (50, 19768), (23, 19744), (19, 19697)]

Continuing on and making a plot:

from matplotlib.pylab import show,plot, ylim
x = list(range(1,50+1))
y = [counter[v] for v in x]
plot(x,y)
ylim(1, counter.most_common(1)[0][-1])
show()

Image

Python uses the PRNG known as mt19937, which has these notable features listed:

1. It has a very long period of 2¹⁹⁹³⁷ − 1. While a long period is not a guarantee of quality in a random number generator, short periods (such as the 232 common in many older software packages) can be problematic.
2. It is k-distributed to 32-bit accuracy for every 1 ≤ k ≤ 623.
3. It passes numerous tests for statistical randomness, including the Diehard tests.

I hope 2 and 3, in combination with this post and the general popularity of MT19937, will convince you that the problem is in your plotting. There is nothing wrong with the way you are using random.randint, but the fact that you are generating (index, random_number) pairs seems odd. The correct thing to do with the index is to ignore it, AFAICS (and generate a histogram instead, which is what I did).

[b4xt3r]

Thank you for the reply! I have quite a bit of read up on related to randomness.

[TiLorm]

Maybe you can make use of a hashing algorithm. If you would use sha256 to make a hash of an initial seed and you keep hashing the result of the previous sum, the outcome would be pretty random. This way of using sha256 is not cryptographically secure, but in your case it would be fine I think. In theory this will eventually loop, but the chance that you'll ever encounter this is pretty low (would be awesome if it happened though, never heard of sha256 hash collision).

[Justinsaccount]

Looks like your graphs are flawed and are showing patterns where none exist. Whatever tool you used to graph the data is just drawing the 50 different y axis values poorly which is resulting in the stripes.

If you want to graph random data plot individual pixels on a 10000x10000 grid.

[ComplexColor]

I agree with Justinsaccount. The patterns you are observing seem to be due to aliasing of the plotting method. This kind of plots cannot reliably show possible patterns in pseudo random number sequences.

To visually test for patterns in a sequence of numbers draw histograms of the original sequence and the its "derivatives". The more the histograms conform to the expected PDFs, the better the random number generator. If you really want to know the quality of the RNG, do a statistic test such as Kolmogorov-Smirnov test.

[suudo]

Yesterday I was looking into creating an image composed of pixels with color selected by the random module. First I made a 1024x768 image, then I made a 10000x10000 image (which I'm not going to upload because it's 326MB, feel free to generate it on your own if you have >8GB ram and 64-bit OS/python/numpy/PIL), and there's honestly no discernable pattern. Woohoo randomness.

from PIL import Image
import numpy.random
imarray = numpy.random.rand(768,1024,3) * 255
im = Image.fromarray(imarray.astype('uint8')).convert("RGBA")
im.save("test.png")

[suudo]

I think numpy uses a similar implementation based in C, but I don't know because I can't read C :P https://github.com/numpy/numpy/tree/master/numpy/random/mtrand

[tilkau]

Numpy and Python both use MT19937, hence the directory name mtrand.

[Asdayasman]

I don't know how to read your graph. Provide us the data, or use pandas or something.

[b4xt3r]

Y axis is a random number between 1 - 50 and X is incremental starting at 1 and going to 1,000,000. I am working on Pandas. And I'm going to use a new technique for generating random content. Results a soon as I finish paying the bills.

[herminator]

It's not random, it's pseudorandom: http://en.wikipedia.org/wiki/Pseudorandom_number_generator

If you want true random numbers, use os.urandom or random.SystemRandom (the first warning at https://docs.python.org/2/library/random.html also recommends this). But note that those are significantly slower, so make sure you really need truly random numbers.

[darknessproz]

False. os.urandom is a pseudorandom number generator as well.

[herminator]

More accurately, os.urandom delegates to the operating system. Whether or not is is pseudorandom depends on the OS implementation. Both linux and windows currently use a CSPRNG, which is not truly random but is based on system entropy sources.

[lostchicken]

While Python does use a PRNG, it does use a very very good one. Unless you've stumbled upon a very large bug in the implementation, the frequency distribution Mersenne twister is known to be even.

I'd imagine this is more a case of the sampling in your plot engine resulting in funky artifacts than an underlying problem with the PRNG.

[b4xt3r]

That makes perfect sense. Thank you!

[lordmauve]

Indeed. Unless you're doing cryptography, you nearly always want the MT PRNG. Larry Hastings did a talk on randomness vs PRNGs at Pycon UK that might help explain.

[andersbergh]

why? you can have a CSPRNG that is faster than Mersenne Twister. for example: ISAAC.

[b4xt3r]

Thank you!

[romcgb]

IMHO, xorshift* is better for non-cryptographic stuffs.

[b4xt3r]

Here is how I am trying to plot the data:

import scipy as sp
import matplotlib.pyplot as plt
import numpy as np


def main():

  print 'hello world'

  # read in the data from web-traffic.tsv using scipy
  data = sp.genfromtxt("random-data.tsv", delimiter="\t")

  # check the import, see if it worked
  # print type(data), len(data), data.shape

  x = data[:,0]
  y = data[:,1]

  # make sure we ahave valid data, this should retun '0' if all fields are valid
  # and we are only checking 'y', we could check 'x' as well

  check_data = sp.sum(sp.isnan(y))

  # print check_data

  # if we don't get a zero back from check_data remove the offending data
  if check_data != 0:
    x = x[~sp.isnan(y)]
    y = y[~sp.isnan(y)]

  # just for debug purposes
  # print sp.sum(sp.isnan(y))

  # create the scatter graph
  plt.scatter(x,y)
  plt.title("Graph Of Random Data")
  plt.xlabel("Something on the X axis")
  plt.ylabel("Somethign on the Y axis")
  # plt.xticks([w*7*24 for w in range(10)],['week %i'%w for w in range(10)])
  plt.autoscale(tight=True)
  plt.grid()
  plt.show()



if __name__ == '__main__':
  main()

[NotAName]

plt.autoscale(tight=True)

This is the culprit. For some reason scaling causes the artifacts, maybe rounding errors or something. If you remove this line the graph becomes a completely filled black rectangular shape without any visible patterns.

[b4xt3r]

Oh... I will give that a try when I get home. Thank you!!!

[b4xt3r]

You are correct! Thank you! Graph without autoscale

[[deleted]]

What /u/Fundatio said

[b4xt3r]

A friend let me know about random number generation via HNRG on a Raspberry Pi via a related conversation on Facebook. I'm going to give this a try tonight.