Overview
ISO 28500:2017 defines the WARC file format - a standardized container for storing harvested Internet content and associated metadata. WARC (Web ARChive) is designed to hold both payload content and control information from mainstream application-layer protocols such as HTTP, DNS, and FTP, and to support large-scale web archiving, digital preservation, exchange and analysis of web resources.
Key Topics
The standard specifies the WARC file and record model and the named fields and record types required for interoperable archives. Important technical topics and requirements include:
- Record model and headers: mandatory and optional fields such as WARC-Record-ID, Content-Length, WARC-Date, WARC-Type, Content-Type, WARC-Target-URI, and digest fields (e.g., WARC-Block-Digest, WARC-Payload-Digest).
- WARC record types: standard types include warcinfo, response, resource, request, metadata, revisit, conversion, and continuation to handle a variety of capture scenarios.
- Control information capture: storage of harvesting protocol control data (for example, request and response headers), not just the response body.
- Metadata linking: associating arbitrary metadata (language, subject classification, encoding) with stored resources.
- Duplicate detection: representing duplicate-detection events linked to resources to reduce storage of identical or highly similar content.
- Data integrity and compression: support for checksums/digests and recommendations for compression to ensure file integrity and efficient storage.
- Extensibility and segmentation: ability to extend WARC without breaking existing tools and to handle overly long records via truncation or segmentation.
Applications
ISO 28500:2017 is essential for organizations and developers involved in:
- Web archiving: national libraries, web archives (e.g., the Internet Archive), and cultural heritage organizations preserving web content.
- Digital preservation: repositories managing born-digital or digitized collections packaged into WARC containers.
- Harvest and crawl tooling: developers building web crawlers, harvesters and replay systems that produce or consume WARC files.
- Data exchange and research: sharing large collections for mining, analytics, and scholarly research while preserving provenance and integrity.
Practical benefits include standardized exchange of large collections, consistent capture of protocol metadata, reduced redundancy through revisit records, and robust long-term storage strategies.
Related Standards
- ARC (legacy format): WARC extends and differentiates from the older ARC format to ensure backward compatibility and clear detection between formats.
- ISO/TC 46: the technical committee responsible for the development of this information and documentation standard.
Keywords: ISO 28500:2017, WARC file format, web archiving, WARC records, digital preservation, web crawler, WARC-Record-ID, WARC-Type, metadata, compression.
