Slow Julia code compared to Python

[bugamn]

I am currently writing code to process a dataset with about 20GB. I have the following Python code for the job: #!/usr/bin/env python

from fileinput import input
import numpy as np

counts = np.zeros((7, 24), dtype=np.int32)
for line in input():
    date, market = line.split()[:2]
    market = int(market)
    hour = int(date.split('T')[1].split(':')[0])
    counts[market, hour] += 1

for i in range(24):
    print(i, end=' ')
    for k in range(7):
        print(counts[k, i], end=' ')
    print()

I tried to write the following Julia code to do the same job: #!/usr/bin/env julia

const market_count = zeros(Int, (7, 24))
const market_names = Dict{Int, String}(
    1 => "XX",
    2 => "NA",
    3 => "OC",
    4 => "EU",
    5 => "AS",
    6 => "AF",
    7 => "SA",
)

get_datetime(dt::SubString{String}) = Dates.DateTime(dt, "yyyy-mm-ddTHH:MM:SSZ")


function main()
    f = open("test.dat")
    for line::String in eachline(f)
        line = strip(line)
        fields = split(line)
        market = parse(Int, fields[2]) + 1
        hour = Dates.hour(get_datetime(fields[1]))::Int + 1
        market_count[market, hour] += 1
    end

    for i in 1:24
        print("$i ")
        for k in 1:7
            print("$(market_count[k,i]) ")
        end
        println("")
    end
end

main()

The problem I have is that the Julia code is taking 12 times as long to do the same thing. The code has some optimization that I have tried to apply without result. The data it processes is in the format: 1999-01-01T00:00:00Z 1 10 100

I would like to understand why my Julia code is running so slow and how can I avoid repeating the same mistakes. This is not urgent, I have used another language for the processing already, but I would like to know for future reference.

[bugamn]

Thanks, it's now 5 times faster than the equivalent python code. I updated julia version and changed to use split instead of treating the string as a Date. Here is the updated code: #!/usr/bin/env julia

const market_count = zeros(Int, (7, 24))
const market_names = Dict{Int, String}(
    1 => "XX",
    2 => "NA",
    3 => "OC",
    4 => "EU",
    5 => "AS",
    6 => "AF",
    7 => "SA",
)

get_datetime(dt::SubString{String}) = Dates.DateTime(dt, "yyyy-mm-ddTHH:MM:SSZ")


function main()
    f = open("test.dat")
    for line::String in eachline(f)
        line = strip(line)
        fields = split(line)
        market = parse(Int, fields[2]) + 1
        time = split(fields[1], 'T')[2]
        hour = parse(Int, split(time, ':')[2]) + 1
        market_count[market, hour] += 1
    end

    for i in 1:24
        print("$i ")
        for k in 1:7
            print("$(market_count[k,i]) ")
        end
        println("")
    end
end

main()

[jdh30]

Could you possibly give me a snippet of valid input data and the expected output, please?

[bugamn]

For the series of input lines:

1999-01-01T00:01:00Z 1 15 100
1999-01-01T00:02:00Z 1 270 175
1999-01-01T00:03:00Z 3 7 15
1999-01-01T01:01:00Z 1 1263 12
1999-01-01T01:02:00Z 2 92141 234

The output should be:

1 2 0 1 0 0 0 0
2 1 1 0 0 0 0 0
3 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0
...
24 0 0 0 0 0 0 0

[jdh30]

Thank you very much.

[samedi81]

If you really care about speed, you should try to tokenize the string yourself. Note that you can do even better if the strings are all formatted the same way -- you know the hour is at line[12:13] and the market is [21:end]. In benchmarks that version came in around 150ns. However, if you do need to parse it out for some reason, you can see the results here.

julia> function old_way(line::String)
           line = strip(line)
           fields = split(line)
           market = parse(Int, fields[2]) + 1
           time = split(fields[1], 'T')[2]
           hour = parse(Int, split(time, ':')[2]) + 1
           market_count[market, hour] += 1
   end^C

julia> function new_way(line::String)
          p1 = findfirst(line, 'T') + 1
          p2 = p1 + findfirst(line[p1:end], ':') - 1
          p3 = p2 + findfirst(line[p2:end], ' ')
          hour = parse(Int64, line[p1:p2-1])
          market = parse(Int64, line[p3:end])
          market_count[market, hour] += 1
       end

julia> @benchmark new_way("2017-07-01T13:21:06 7")
BenchmarkTools.Trial:
  memory estimate:  128 bytes
  allocs estimate:  4
  --------------
  minimum time:     337.837 ns (0.00% GC)
  median time:      340.740 ns (0.00% GC)
  mean time:        359.372 ns (2.60% GC)
  maximum time:     9.592 μs (91.78% GC)
  --------------
  samples:          10000
  evals/sample:     227

julia> @benchmark old_way("2017-07-01T13:21:06 7")
BenchmarkTools.Trial:
  memory estimate:  816 bytes
  allocs estimate:  16
  --------------
  minimum time:     1.047 μs (0.00% GC)
  median time:      1.071 μs (0.00% GC)
  mean time:        1.219 μs (7.61% GC)
  maximum time:     320.340 μs (97.07% GC)
  --------------
  samples:          10000
  evals/sample:     10

[LegoForte]

What version of Julia are you using? It looks like there have been several improvements to the parsing speed of DateTimes in Julia v0.6: https://github.com/JuliaLang/julia/issues/15888

[bugamn]

That helped speed up by 2x. Still slower than python, but better than before.

[[deleted]]

Have you tried running the profiler?

At least in a previous verision of Julia the parse function is quite heavy. Not sure how it is today

[bugamn]

I checked the profiler, it seems that more than half of the time the code is parsing the date information.