Image Processing

We use Python’s Process Pool Executor for parallel image processing in background worker processes. This allows multiple images to be processed in parallel without blocking the API or frontend.

PictoPy uses different models for achieving its tagging capabilities. The discussed models below are default models, you can change them by going to app/models directory and change the paths in the configuration files.

Object Detection with YOLOv11

We use YOLOv11 to spot objects in your photos. Here's what it does:

YOLOv11 takes your image and runs it through its model. It figures out what objects are in the image and where they are. The result is a list of objects, their locations, and how confident the model is about each detection. If a person class is predicted we pass it on to the face detection model which we discuss in the next section.

Fun Fact

YOLO stands for "You Only Look Once". We use the model provided by Ultralytics by default.

Face Detection and Recognition

For faces, we do a bit more:

We start with a special version of YOLOv11 that's really good at finding faces. Once we find a face, we zoom in on it (by cropping it to 160x160 - the shape FaceNet expects) and pass it to our FaceNet model. FaceNet then creates a unique 'embedding' for each face, the representation of the face in a form of numbers.

Fun Fact

We use another YOLOv11 model for this as well by default. This was pretrained on top of the one provided by Ultralytics and is called yolov11-face

What's an embedding?

An embedding is a bunch of numbers that represent the face. Similar faces will have similar numbers. FaceNet creates a 512-dimensional embedding array for each detected face in the image.

Face Clustering

Now, here's where it gets interesting:

We use something called DBSCAN to group similar faces together. This process happens automatically as you add new photos to the system, we perform reclustering after every 5 photos are added (this can be changed in the code) but apart from that, the photos are assigned a cluster based on the embedding distance of the faces in the photo with the mean of each of the clusters.

Semantic Search with SigLIP2

Beyond tag-based search (finding photos by the exact object/face labels YOLO and FaceNet detected), PictoPy can also search photos by describing them in plain language — "beach sunset", "two people hugging" — using Google's SigLIP2 model.

Every photo gets a single numeric "embedding" computed once, in the background, right after the usual object/face tagging pass finishes. When you search, your query gets embedded the same way and compared against every stored photo embedding — so a phrase the app has never seen before still works immediately, with no re-scan of your library required.

Just type into the same search box you already use. There's no separate "semantic search" mode to switch on: PictoPy tries an exact tag match first, and only falls back to meaning-based search if that comes up empty (and the feature is installed).

Fun Fact

SigLIP2 stands for "Sigmoid Loss for Language-Image Pre-training, v2". Unlike its predecessor CLIP, it uses a per-pair sigmoid loss instead of a whole-batch softmax during training — this is why its match scores look different from what you might expect (see the parameters table below).

Installing it

Semantic search is an optional ~1.5 GB download from Settings → AI Models, listed as a "Semantic Search" bundle (three files: a vision model, a text model, and a tokenizer). If it isn't installed, tag search keeps working exactly as before — semantic search just silently doesn't contribute any results.

For the full technical breakdown — architecture diagrams, database schema, model calibration details, and known limitations — see the dedicated Semantic Search page.

How It All Fits Together

When you add a new photo, we first look for objects and faces. If we find faces, we generate embeddings for them. These embeddings then get added to our face clusters. Then, if the semantic search models are installed, we generate a SigLIP2 embedding for the photo too. All this information gets stored in our database so we can find it later.

Under the Hood

We're using ONNX runtime to run our AI models quickly. Everything's stored in SQLite databases, making it easy to manage. The system updates clusters as you add or remove photos, so it keeps getting smarter over time.

PictoPy Model Parameters

Here are some key parameters for the main models used in PictoPy's image processing pipeline.

YOLOv11 Object Detection

Parameter	Value	Description
`conf_thres`	0.4	Confidence threshold for object detection
`iou_thres`	0.5	IoU (Intersection over Union) threshold for NMS
Input Shape	Varies	Determined dynamically from the model
Output	Multiple	Includes bounding boxes, scores, and class IDs

Face Detection (YOLOv11 variant)

Parameter	Value	Description
`conf_thres`	0.35	Confidence threshold for face detection
`iou_thres`	0.45	IoU threshold for NMS in face detection
Model Path	`DEFAULT_FACE_DETECTION_MODEL`	Path to the face detection model file

FaceNet (Face Recognition)

Parameter	Value	Description
Model Path	`DEFAULT_FACENET_MODEL`	Path to the FaceNet model file
Input Shape	(1, 3, 160, 160)	Expected input shape for face images
Output	512-dimensional vector	Face embedding dimension

Face Clustering (DBSCAN)

Parameter	Value	Description
`eps`	0.3	Maximum distance between two samples for them to be considered as in the same neighborhood
`min_samples`	2	Number of samples in a neighborhood for a point to be considered as a core point
`metric`	"cosine"	Distance metric used for clustering

Semantic Search (SigLIP2)

Parameter	Value	Description
Default checkpoint	`base`	Set via `SIGLIP2_ACTIVE_CHECKPOINT`; `large` and `so400m` also exist but ship placeholder registry entries only (see Semantic Search).
Input resolution (`base`)	224 × 224	Larger checkpoints use 384 × 384.
Embedding dimension	768	Same dimensionality for both the image and text towers.
`SIGLIP2_EMBED_BATCH_SIZE`	8	Images per batch during the background embedding pass.
`SIGLIP2_MATCH_THRESHOLD`	0.01	Minimum sigmoid score to count as a match. SigLIP2's absolute scores run low even for real matches — this is expected, not a bug.
Output	Sorted, scored image list	Scores are rounded to 4 decimal places server-side and never shown in the UI.

Note: Some of these values are default parameters and can be adjusted when initializing the models or during runtime, depending on the specific use case or performance requirements.