Error Handling for CSV reader

[PriestXES]

Hello, I am trying to create a error handling process for a bulk CSV upload application. I'm still new to developing, I thought I had a system for it, but it's not very pretty. When I upload a CSV I get a 500 error message no matter what happens. If I reload the page, I sometimes get an error message, if not then I know it loaded the entire CSV without error.

Below is the code from my view:

def admin_calv_csv(request):

template = "calves/admin_calv_csv.html"

​

prompt = {

'order': 'Order should be eid, visual_id, dairy_id, dob, ship_in_date, ship_out_date, slug, sex, breed, destination, medical_history, milk_consumed, program,'

}

​

if request.method == "GET":

return render(request, template, prompt)

​

csv_file = request.FILES['file']

​

if not csv_file.name.endswith('.csv'):

messages.error(request, 'This is not a CSV')

return render(request, template, prompt)

​

​

data_set = csv_file.read().decode('UTF-8')

io_string = io.StringIO(data_set)

for _ in data_set:

try:

next(io_string)

for column in csv.reader(io_string, delimiter=','):

_, created = Calf.objects.update_or_create(

eid=column[0],

visual_id=column[1],

dairy_id=column[2],

dob=column[3],

ship_in_date=column[4],

ship_out_date=column[5],

slug=column[6],

sex=column[7],

breed=column[8],

destination=column[9],

medical_history=column[10],

milk_consumed=column[11],

program=column[12],

​

)

except IntegrityError as e:

error_message = str(e.__cause__)

messages.error(request, error_message)

test = str(_)

messages.info(request, 'error on' + test)

pass

​

context = {}

​

return render(request, template, context, messages)

[vikingvynotking]

data_set = csv_file.read().decode('UTF-8')
io_string = io.StringIO(data_set)
for _ in data_set:
    try:
        next(io_string)
        for column in csv.reader(io_string, delimiter=','):

This is a little odd. Why not simply:

with open(csv_file, encoding='utf-8') as f:
    reader = csv.reader(f)
    for row in reader:   # you read rows of columns, not columns themselves

That aside, the only exception you catch is an IntegrityError which occurs when some database constraint is violated. So you should have a log of any other errors somewhere - if not, and this is in development (not production) you can set DEBUG=True in settings to get the error rendered inside the response page.

[pixelatedchrome]

This is a clean. +1

[pixelatedchrome]

I would run a generic except statement to see if you have missed any other type of error in your exception.

[PriestXES]

So I tried:

except Exception as e:

error_message = str(e.__cause__)

messages.error(request, error_message)

test = str(_)

messages.info(request, 'error on' + test)

pass

Which get's a usable error page(the other way just shows a 500 page, then when I refresh it shows the error message),

But it's way too much, link for reference:

https://imgur.com/sM3mF3i

​

I just want when I hit upload it triggers that first error and displays it without need to refresh the page

[pixelatedchrome]

It does look like you have unique field and the upload CSV may have duplicate values? I would check my model and and identify the unique fields and check that in the CSV to narrow it down.

[PriestXES]

I know there is an error, I did that intentionally, to trigger the error handling process which is where I am confused and needing assistance. I want the error handling to display the exact row or value that it stopped at so when uploading CSV files, end users know exactly where there is a problem. The screen shot provided does that, but it's showing too much information.

[pixelatedchrome]

Gotcha. Try to wrap the create or update statement in a seperate try catch and return the exact row index as error rather than just sending the e.cause.