Practical guide

How to automate product imports into an e-commerce shop

Let the import run on its own, but do not leave it unattended. Check and measure data, and make every run safe to repeat.

35 minutes · XML, CSV, and automation

First, the short version

What does an automated import solve?

A supplier usually gives you a regular file or URL containing products. The import downloads it, reads it, checks it, and writes changes into the shop without manual transcription.

XML and CSV are common formats for such a data feed. The format itself does not guarantee that data makes sense. Your import must check that.

Get ready

What to decide before the first automation

Set the rules first. Then write a command that can repeat them.

A sample file or test URL from the supplier. Never start with production data without a way back.
A unique product identifier, such as a SKU or supplier code. A product name is usually not enough.
A decision on which data the supplier can overwrite and which data the shop owns. Price and stock may come from a feed; a description might not.
A time window for the import and contact details for someone who understands the supplier data.

Steps 1 to 3

Split the import into three safe parts

Every step must be traceable. When something fails, you want to know where and for which product.

1. Download and verify the source

Download the feed over a secure connection and set a timeout. Do not wait forever for a third-party server.
Store the date, file size, and, where available, its checksum. You can later see whether new data arrived.
Check that the file truly contains expected XML or CSV. You must not parse an HTML error page as a product feed.
If the source fails a check, stop the import. Older correct data is better than quickly saved bad data.

Official Symfony HTTP download documentation

2. Normalise and write in batches

Map each line or item to your own shape: external code, name, price, stock, and status.
Check required fields, numbers, and currency. Skip an invalid item, record it, and continue.
Look up a product by its stable external code. Then create or update it.
Write in smaller batches. You do not want to keep a large file in memory or lock the database for a long time.

php bin/console app:products:import --source=/path/to/feed --dry-run

Official Symfony console command documentation

3. Schedule the run and watch the result

First run the command manually and in dry-run mode. Only then give it to a scheduler such as cron.
One run must not start a second run of the same import. Use a lock or put the import into a message queue.
After finishing, record the number of new, changed, skipped, and failed products.
Send an alert when a full run fails or the error count rises. Silence does not mean the import works.

php bin/console app:products:import --source=https://supplier.example/feed.xml

Official Symfony background-work documentation

Step 4

Check the import on a small sample

First inspect a few items manually. Then you can run the whole feed with much more confidence.

Run a dry run

The command should show what it would create or change, but it must not write to the database. This mode is also useful when investigating failures.
```
php bin/console app:products:import --source=/path/to/feed --dry-run
```
Check three different products

Choose a new product, a changed product, and an item with an empty value. Compare the code, price, stock, and shop visibility.
Run the import a second time

With the same feed, the second run should not create duplicates. At most, it should confirm that data is already current.
```
php bin/console app:products:import --source=/path/to/feed
```

If something goes wrong

Common problems

The feed downloads, but it is not the expected file

The supplier may have returned a login page, error page, or empty content. Check the HTTP status, content type, size, and the first parser error.

The import creates duplicates

The cause is often an unstable identifier. Link a product using a permanent external code and add a unique database constraint.

One bad product stops the whole run

Record the error for that item and continue. Stop the whole import only when the source is invalid or you cannot safely decide what to write.

Two imports run at the same time

Add a lock for the entire run. The second process should wait or stop with a clear message, not write the same products concurrently.

Done

The import can work on its own.

Now run it regularly, but watch summaries and errors. A reliable import is not one that never reports a problem; it is one where you find and safely fix a problem quickly.

Read XML in the glossary All guides