ETL stands for extract, transform, load: a data workflow that pulls information from one or more sources, reshapes or cleans it, then writes it to a destination. In a scraping project, extraction may collect page fields, transformation may normalize dates and remove duplicates, and loading may store records in a database or file.
Keep each stage traceable
Preserve enough source information to tell what the transformation changed. For example, retain the original price string as well as a parsed numeric value and currency. Validate records before loading, and route failures somewhere visible instead of silently dropping them.
ETL is often contrasted with ELT, where raw data is loaded first and transformed in the destination system. Either can work. Choose based on data volume, destination capabilities, and whether the raw source needs to remain available for reprocessing.
Retries deserve attention: loading the same batch twice should not accidentally duplicate all rows. Use stable keys or idempotent upserts where appropriate, and record batch identifiers and run times.
Leave a Reply