Checksums and data integrity

01Keep a reference to what the bytes should be

A file service sends a 100 MB report in 100 chunks of 1 MB. How can the receiver detect a changed chunk rather than merely accept whatever bytes arrive?

1. The sender computes a compact value from the intended bytes using a defined algorithm.
2. The receiver gets the expected value from the intended file service over a TLS connection that verifies that service’s identity, rather than accepting a replacement value bundled with an untrusted download.
3. The receiver computes the value again from the downloaded bytes.
4. It compares the values. A mismatch means the bytes and reference disagree.

That compact value is a checksum, or a digest when discussing a hash calculation. A digest has a fixed output size even when the file is large. A mismatch does not reveal the cause and cannot reconstruct the original bytes.

For the report, obtain a trusted manifest from that service: a list of chunk positions, expected lengths, and expected digests, plus the named digest algorithm and complete-file digest. The positions matter because correct chunks in the wrong order can still form the wrong report.

Choose the method according to the threat:

• A cyclic redundancy check, or CRC, efficiently detects common accidental changes under its error model.
• A cryptographic hash is designed to make deliberate collision-finding difficult under its security assumptions.
• A collision means different byte sequences produce the same output. Finite outputs mean collisions remain possible; a matching digest is evidence under the chosen method, not a mathematical unique-file guarantee.

02Separate corruption detection from authenticity

An attacker replaces a file and also replaces the plain hash published beside it. The receiver recomputes the replacement’s hash and sees a match. The comparison worked, but its reference was untrusted.

Separate two questions:

• Integrity asks whether the bytes agree with the expected reference.
• Authenticity asks whether the message or reference came from the intended source.

Use a trusted reference or an established authentication method:

1. With HMAC, sender and verifier share a secret key. The sender uses the standard algorithm to produce a message authentication code; the verifier uses the shared secret to check it. Someone without the key should not be able to create a valid code for altered data.
2. With a digital signature, the sender signs using a private key. The receiver verifies using the corresponding public key and the chosen standard scheme.
3. The receiver must know that the public key belongs to the intended signer. Accepting an attacker’s replacement file, signature, and replacement key does not establish the original source.

Use established schemes and key-management rules. Inventing a hash-plus-secret formula is not a substitute for them.

A digest can also name stored content. This is content-addressed storage:

• Recomputing a trusted content identifier helps detect unexpected returned bytes.
• Matching content identifiers can help identify duplicate files and reuse stored bytes.
• The identifier does not decide who may read the file, whether the content is safe, or whether the uploader owns it.

Byte checking, source authentication, and access permission remain separate decisions.

03Detect the problem, then recover safely

Verify and repair the chunked report in order:

1. Obtain the intended manifest from the authenticated file-service connection, or verify it using the HMAC/signature method described above. Do not trust a new manifest merely because it arrived beside the suspect bytes.
2. Download a chunk and compute its digest using the specified algorithm.
3. Compare it with the digest assigned to that position, and verify the expected length.
4. If chunk 37 mismatches, reject or quarantine those bytes. Request a replacement from a source whose bytes can pass the same trusted check.
5. Place every verified chunk in its assigned position.
6. Check the assembled file against its complete-file digest when provided.

In the simple retry model, replacing one bad 1 MB chunk avoids retransmitting the other 99 MB. Checking the entire assembly additionally helps detect wrong ordering or assembly mistakes.

Handle failure honestly:

• Do not replace the expected digest with the newly calculated one just to silence an error. That declares the suspect bytes correct without evidence.
• Another replica is useful only when its bytes can be verified against the trusted reference. “Different machine” alone does not establish correctness.
• If no verified replacement exists, report the failure rather than accepting unknown bytes.

Periodically reading stored data and comparing its expected digest is called background scrubbing. It can detect damage in a report nobody has opened recently. Repair still needs verified replacement bytes.

Monitor mismatches by storage location and record repair failures. The complete design includes an expected reference, a comparison, and a recovery action; calculating a hash supplies only part of that design.