This post is part of a multi-part series documenting the architecture, design decisions, and trade-offs behind building Yaldi.chat.

Handling Image Metadata, Fingerprinting, and Search Beyond SQL

Beyond just imges, emotes are symbols, variants, memes, and iterations. They get recolored, resized, cropped, mirrored, animated, re-uploaded, and renamed. This post goes deep into why Yaldi records rich emote metadata, how fingerprinting works, and how search and discovery extend beyond traditional SQL queries.

The Problem: Visual Duplication Is Not Binary

Users want Yaldi to allow for uniqueness to their community while also having a non-cluttered and redundant searching UX. In parallel, I need Yaldi to maintain a low-cost TCO. Two emotes can be:

  • Byte-for-byte identical
  • Visually identical but encoded differently
  • Nearly identical with small edits
  • Clearly derived from the same source
  • Completely unrelated

A filename or checksum alone can’t express those relationships. Yaldi treats emotes as visual data, not just files. This approach helps eliminate duplicative files bloating the cost of the platform or ruining the user experience with duplicative noise.

Capturing Metadata at Upload Time

Metadata is cheapest to compute when the file first enters the system. At upload time, Yaldi performs a series of inspections and transformations using Python imaging libraries (primarily Pillow) and hashing algorithms. This allows expensive or complex analysis to happen once, instead of repeatedly at query time. The only trade off is compute for evaluations of images which may never be used or used infrequently and a slightly slower UX (a couple of seconds) when the image is first uploaded while the evaluations are performed. We avoid the slower UX by running some of these evaluations in parallel after the initial upload is complete (e.g. while the user is giving a name to the emote). Below outlines the general flow of evaluations.

Byte-Level Hashing: SHA-256

The first fingerprint is a SHA-256 hash of the raw file bytes. This allows us to quickly answer "is this file exactly the same as another file we’ve already seen?" without wasting additional resources to evaluate further. This also allows for quick quality checks to weed out trash files and abuse.

If two emotes share the same SHA-256 hash, they are identical at the byte level—no ambiguity. But this only catches the simplest case and misses other issues like same image but different file format, for example.

Perceptual Hashing: Seeing Like a Human

To detect visual similarity, Yaldi computes multiple perceptual hashes using Pillow-based algorithms, each having strengths and weaknesses. Using multiple hashes provides a more reliable similarity signal than relying on just one:

  • pHash - perceptual hash - more robust to scaling and compression artifacts
  • aHash - average hash - simple and fast, but sensitive to brightness changes
  • dHash - difference hash - captures gradients and edges more effectively

These hashes are designed to change slowly as an image changes visually. Two images that look similar will have hashes with a small Hamming distance. Before computing perceptual hashes, Yaldi normalizes images by converting them to a consistent color space, resizing to a fixed resolution, stripping metadata, and normalizing aspect ratio (where possible). This ensures that hashes reflect visual content, not incidental encoding differences.

Animated Emotes: A Harder Problem

Animated emotes complicate everything. Yaldi handles animated images by:

  1. Extracting representative frames
  2. Computing perceptual hashes per frame
  3. Aggregating or sampling hashes

This allows detection of animated variants derived from the same source—even if timing or frame count differs.

What Yaldi Does With These Hashes

1) Duplicate detection at upload

Exact duplicates (SHA-256 matches) can be flagged immediately. Near-duplicates (small perceptual distance) can be surfaced for review or redirection use the existing emote for the user:

 

2) Related emote discovery

When viewing an emote, Yaldi can quickly query for visually similar emotes by comparing perceptual hashes within a configurable distance threshold. This enables features like:

  • “Related emotes” suggestions
  • Variant grouping
  • Source lineage exploration

 

3) Abuse and spam detection

Repeated uploads of slightly modified emotes are easy to detect when perceptual hashes cluster tightly. This provides a moderation signal that doesn’t rely on filenames or user behavior alone.

4) Future-proofing search and recommendations

Hash-based similarity creates a foundation for recommendation systems, clustering, and discovery that can evolve without changing the upload pipeline.

Emote Search: Why SQL Alone Is Not Enough

SQL is excellent at answering precise questions of who owns the emote, where is it being uses, and what's the basic metadata. It is not designed for fuzzy visual similarity. Yaldi treats search as a multi-stage process:

  1. Use SQL to narrow the candidate set
  2. Apply hash-based similarity scoring
  3. Rank and filter results in application logic

Indexing Strategies for Hash Data

Hash values are stored explicitly and indexed for fast retrieval. SQL handles exact hash lookups, range queries on numeric hash representations, and candidate filtering by metadata.

Hamming distance calculations and ranking happen outside the database where flexibility is higher and performance is predictable.

Performance Considerations

Computing perceptual hashes is relatively expensive. Comparing them at scale is even more so. Yaldi avoids performance pitfalls by:

  • Computing hashes once, at upload time
  • Limiting similarity queries to narrowed candidate sets
  • Using configurable distance thresholds
  • Avoiding full-table scans for similarity operations

The database remains a fast index—not a compute engine. This system is extensible. Hashing and metadata provide a bridge toward machine learning classification, semantic tagging, visual clustering, and smarter recommendations