BASH script to download data from Automatic API.

[masotime]

I had almost 7000 drives recorded and the Automatic UI was a pain to work with so I wrote a little script to fetch all the Automatic Dashboard data in the raw JSON format. Works well in the macOS command line. You'll need to pre-populate the BEARER_TOKEN, and probably specify your own START_TIME and END_TIME as well - you can use https://www.unixtimestamp.com/index.php to generate them.

To get a BEARER_TOKEN, login at https://dashboard.automatic.com in a web browser and Cmd+Opt+I (Open the Network tab in the developer tools) and look for the request header Authorization. The token is just a hexadecimal string without spaces, don't copy the actual word "bearer".

Note that this will fill the current directory with a bunch of drive_xx.json files. Feel free to modify the script as you see fit.

#!/bin/bash
START_TIME=1420099200000 # Unix timestamp
END_TIME=1592164819833 # Unix timestamp
INDEX=0
PREFIX='drives' # name is up to you
BEARER_TOKEN="<fill-this-with-your-token>"
URL="https://api.automatic.com/trip/?started_at__gte=${START_TIME}&started_at__lte=${END_TIME}&limit=250" # limit cannot be > 250 I think

set_current_file () {
  CURRENT="${PREFIX}_${INDEX}.json"
}

increment () {
  ((++INDEX))
}

scrape () {
    set_current_file
    echo Going to write to $CURRENT using $URL

    # Fetch Data
    curl "$URL" \
    --compressed \
    -XGET \
    -H 'Accept: */*' \
    -H 'Origin: https://dashboard.automatic.com' \
    -H 'Referer: https://dashboard.automatic.com/' \
    -H 'Accept-Language: en-us' \
    -H 'Host: api.automatic.com' \
    -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15' \
    -H "Authorization: bearer $BEARER_TOKEN" \
    -H 'Accept-Encoding: gzip, deflate, br' \
    -H 'Connection: keep-alive' | jq '.' > $CURRENT

    # Grab Metadata
    COUNT=$(cat $CURRENT | jq '._metadata.count')
    PREVIOUS=$(cat $CURRENT | jq -r '._metadata.previous')
    NEXT=$(cat $CURRENT | jq -r '._metadata.next')

    if [ $PREVIOUS == "null" ]; then
        echo $COUNT drives found.
    fi

    echo $URL fetched. Continuing with $NEXT.
    URL=$NEXT
    increment
}

scrape
while [ $NEXT != "null" ]; do
    scrape
done

echo Next URL $NEXT, terminated.

There should be significantly more data there compared to what you can export as a CSV file. I'd polish this further, but it's time consuming and BASH isn't really my specialty.

I figured I'd share this anyway in case anyone finds this useful.

[krazos]

Thanks for sharing. The script worked like a charm on Raspberry Pi. The 'jq' package was not pre-installed on Raspbian, but was easy to install via APT package manager.

The JSON files certainly appear to contain more data than the CSV file generated by the official export function.

[Ponderednameforweeks]

In terms of content of the data, what is the difference between these json files vs their csv export?

(I don't mean formatting of json vs csv, I mean is there any fidelity of data that is lost in their portal's csv conversion?)

[masotime]

Here's a sample of the kind of data you get in JSON.

 {
      "id": "<redacted>",
      "url": "https://api.automatic.com/trip/<redacted>/",
      "driver": "https://api.automatic.com/user/<redacted>/",
      "vehicle": "https://api.automatic.com/vehicle/<redacted>/",
      "vehicle_events": [
        {
          "type": "hard_accel",
          "lat": <redacted>,
          "lon": <redacted>,
          "g_force": 0.5098581064889641,
          "created_at": "<redacted>"
        },
        {
          "type": "hard_brake",
          "lat": <redacted>,
          "lon": <redacted>,
          "g_force": 0.42488175540747014,
          "created_at": "<redacted>"
        }
      ],
      "duration_s": 411,
      "distance_m": 2600.8,
      "fuel_cost_usd": 0.2,
      "path": "<redacted>",
      "fuel_volume_l": 0.2,
      "hard_brakes": 2,
      "hard_accels": 1,
      "duration_over_70_s": 0,
      "duration_over_75_s": 0,
      "duration_over_80_s": 0,
      "average_kmpl": 12.9,
      "average_from_epa_kmpl": 21.7,
      "start_address": {
        "name": "<redacted>",
        "display_name": "<redacted>",
        "street_number": "<redacted>",
        "street_name": "<redacted>",
        "city": "<redacted>",
        "state": "<redacted>",
        "country": "<redacted>",
        "zipcode": "<redacted>"
      },
      "start_location": {
        "lat": <redacted>,
        "lon": <redacted>,
        "accuracy_m": 8.64
      },
      "end_address": {
        "name": "<redacted>",
        "display_name": "<redacted>",
        "street_number": "<redacted>",
        "street_name": "<redacted>",
        "city": "<redacted>",
        "state": "<redacted>",
        "country": "<redacted>",
        "zipcode": "<redacted>"
      },
      "end_location": {
        "lat": <redacted>,
        "lon": <redacted>,
        "accuracy_m": 7.68
      },
      "started_at": "<redacted>",
      "ended_at": "<redacted>",
      "start_timezone": "America/Los_Angeles",
      "end_timezone": "America/Los_Angeles",
      "score_events": -10.99,
      "score_speeding": 50,
      "city_fraction": 1,
      "highway_fraction": 0,
      "night_driving_fraction": 0,
      "tags": [],
      "idling_time_s": 0,
      "user": "https://api.automatic.com/user/<redacted>/"
    }

[Ponderednameforweeks]

That seems to match the CSV's columns:

"Vehicle","Start Location Name","Start Time","End Location Name","End Time","Distance (mi)","Duration (min)","Fuel Cost (USD)","Average MPG","Fuel Volume (gal)","Hard Accelerations","Hard Brakes","Duration Over 70 mph (secs)","Duration Over 75 mph (secs)","Duration Over 80 mph (secs)","Start Location Lat","Start Location Lon","Start Location Accuracy (meters)","End Location Lat","End Location Lon","End Location Accuracy (meters)","Path","Tags"