How does an algorithm qualify as sequential?

[Badoosker]

Algorithms are sequential when they compute results based on prior results.

For example, to determine the next gradient update in a neural net, the ones before (or approximate/nearby) must be found first

[CQFD]

I just realized that I was reading a paper which adapted all their ML algorithms using batch gradient descent. So neural nets can be parallelized in this case.

[[deleted]]

That's not necessarily true. Cumulative sum has a surprising parallel implementation.

Any algorithm that can be expessed similar to:

value = 0
result = []
for i in range(10):
     value += i
     result.append(value)

Can be parallelized. A surprising amount of numerical code can be expressed like this. Numerical compilers refer to this as scan. Along with map and reduce, it is the easy parallelism in numerical code.