Challenge
Developers must submit asking prices of flats to the dane.gov.pl portal every day, and neglecting this duty can lead to a charge of violating the collective interests of consumers. The client generated the files from its website, but generation only ran when the site had traffic, so no file was produced for 11 days.
The files also did not match the Ministry of Digital Affairs template: the KRS number contained a spreadsheet formula, the contact method was missing, parking spaces ended up in the wrong columns, and sold units remained in the data. The portal was missing 112 days in two series and 185 days in the third.
Approach
Audit first: we compared all 612 CSV files and 2 XML files with the act, the ministry template (58 columns), the guidelines and the official XSD schema. We fixed the generator at the source and rewrote the archive retroactively with the same rules, instead of patching files by hand.
The portal offers no write API, and uploading about 400 resources by hand was out of the question. So we wrote a browser-based automation that checks what is already on the portal and skips existing dates. We made a backup before every change, because the archive has to be preserved in full.
Implementation
The generator creates files twice a day from a system scheduler, before the time at which the portal collects them. A validator checks every file against the template and the act, and a control script compares the portal file with the local version byte by byte and makes sure the set of dates is complete and free of duplicates.
One thing could not be reconstructed retroactively: the sale dates that the system never recorded. Those units stayed in the history as they were published, and from the day of deployment the generator omits them correctly.
Results
In a single session we uploaded 409 resources in six parallel processes. Final check: 112 of 112, 112 of 112 and 185 of 185 days matching in the three series, with no gaps and no duplicates.
Validation of the whole archive, 645 files including the backfilled days, passed without errors. The XML file passes the official schema with no remarks. Files are now generated automatically every day, and the next step is registering automatic data collection by the portal.
