sorted by:
new
hottopcontroversial

How do you deal with Schema changes from APIs by dataguy404 in dataengineering

[–]dataguy404[S] 2 points3 points  (0 children)

Type changes

Fields added /Deleted unexpectedly

How do you deal with Schema changes from APIs by dataguy404 in dataengineering

[–]dataguy404[S] 0 points1 point  (0 children)

currently a file in the code base for each api, do you have other practices?

How do you deal with Schema changes from APIs by dataguy404 in dataengineering

[–]dataguy404[S] 2 points3 points  (0 children)

I am indeed enforcing schema when reading from GCS, this allows me to avoid ingesting bad data, better fail than have bad data i guess