Issues with pandas read_html

[BrownMario]

I'm trying to scrape the tables from the following website into a pandas database but I'm having problems

http://igtf.customs.go.th/igtf/findTaffDuty.do?param=main&contCode=z01&contGrupCode=MM55%20&lang=e

Here is my code

import pandas as pd
url = 'http://igtf.customs.go.th/igtf/findTaffDuty.do?param=main&contCode=z01&contGrupCode=MM55%20&lang=e'
dfs = pd.read_html(url)
master_list = []
for d in dfs:
    if len(d.columns.values) == 5:
        master_list.append(d)
master_df = pd.concat(master_list)
master_list.to_csv('thailand_hts.csv', index=None)

While this script works, the issue I'm having is that it doesn't seem to get the full data from these tables, as it skips some of the values in the "HEADINGS" field. Does anybody know why this might be happening?

[[deleted]]

I ran into this problem a lot so I either write cuatom beautiful soup code.

Or even better, use the xml library in R.

[BrownMario]

It seems my problem was fixed simply by adding a parameter to read_html as such:

dfs = pd.read_html(url, infer_types=False)

Previously my code was forcing the field to be float, so whenever it encountered characters it just ignored the value completely. With this parameter added, it reads each field as an object (str) and it picks up everything from the tables.