Jan Mikoláš
Practical guide
How to design a product feed and automate updates
Create a versioned export that consumers never download halfway and whose freshness you can measure.
In short
A feed is a data contract, not a table view
Every field has a defined meaning, type, unit, and missing-value rule. A stable product ID, format version, and generation time let consumers process changes safely.
Write a large XML or CSV feed to a temporary file as a stream. Publish only after completion and validation so consumers never see a work in progress.
Prepare
Define the consumer and SLA
A price comparison site, marketplace, and internal warehouse need different contracts.
- Required and optional fields, including stable ID, variants, price, currency, availability, and URL.
- XML or CSV format, UTF-8 encoding, decimal separator, time zone, and enum values.
- A full or incremental export decision, including how deletions are represented.
- Required frequency, maximum age, download method, authentication, and expected size.
Steps 1 to 3
Generate, validate, and only then publish
Separate domain product selection, channel-specific mapping, and technical file writing.
1. Version an exact format
- Every record has a stable product_id or variant_id. A name or URL is not an identity.
- Serialize prices as exact decimals with currency and times as ISO 8601 with a time zone. State units explicitly.
- Distinguish a missing value, empty text, and zero. Escape XML with a library and write CSV with one agreed delimiter and quoting policy.
- Publish a breaking change under a new endpoint or file version and maintain both contracts during a transition.
<feed version="2" generated_at="2026-08-11T10:00:00+02:00">2. Export in streams and batches
- Read products with stable keyset pagination in batches, not all into memory or with offsets over a changing table.
- Write XML with XMLWriter and CSV with fputcsv. Clear ORM entities after each batch and count rows and checksums incrementally.
- A full feed contains current state. An incremental feed defines a time window, ordering, and tombstone for a deleted product.
- Handle channel differences with adapters over a shared export model, not scattered conditions in one generator.
$writer->startElement("product");3. Publish atomically and monitor cron
- Write to a unique temporary file. Atomically rename it after validation; in object storage use a versioned object and a pointer.
- Publish a checksum, size, row count, and generated_at alongside the feed. Offer gzip, ETag, and Last-Modified where useful.
- Protect cron against overlap with a distributed or database lock. Give each run an ID and bounded execution time.
- Alert on a missed run, excessive age, unusual row drop, validation failure, and generation that takes too long.
php bin/console app:feed:generate productsStep 4
Test the contract and publication
A few clean products are not enough. The feed must handle special characters, deletion, and an interrupted run.
-
Validate boundary data
Test diacritics, ampersands, quotes, CSV newlines, zero price, a missing image, and a long description.
php bin/phpunit --filter ProductFeed -
Interrupt generation halfway
The public URL still serves the last complete feed; the temporary file is cleaned up safely after timeout.
-
Skip a scheduled run
Monitoring based on generated_at alerts, and the next run restores a fresh export without manual replacement.
Troubleshooting
Common problems
The feed exhausts memory
Do not load the full collection or XML document. Read batches and write incrementally with XMLWriter or fputcsv.
Consumers sometimes download a truncated file
You write directly to the public target. Generate elsewhere, validate, then publish atomically.
Deleted products remain at a partner
The incremental contract has no tombstone. Add an explicit delete record or a periodic full snapshot.
Two cron runs overwrite the same feed
Add a lock with a sensible TTL, unique temporary names, and an overlap metric.
Done
The feed is stable, fresh, and published whole.
A versioned contract, streaming export, atomic replacement, and monitoring give consumers predictable data without broken intermediate states.