Facial Emotion Recognition

01The problem

Facial emotion recognition sits behind real products — accessibility tools, UX research, driver monitoring, moderation. The task here: classify a face into happy, sad, neutral or surprise from a 48×48 grayscale image. At that resolution an eyebrow is a few pixels; the difference between sad and neutral can be a shadow.

Two things make this harder than a standard image-classification problem. The input is stripped of almost everything a modern vision model expects — no color, no texture detail, no context around the face, and a total of 2,304 pixels to work with. And the labels themselves are soft: humans disagree about these four categories too, particularly between sad and neutral, which puts a real ceiling on achievable accuracy. A model that hit 99% here would be evidence of a leak, not of skill.

So the interesting question wasn't "can a CNN do this" — it was which kind of model earns its complexity on small, low-resolution, single-channel data.

02The approach

Four architectures went head-to-head under the same training and evaluation protocol:

• VGG16, ResNet and EfficientNet — ImageNet-pretrained backbones adapted via transfer learning, the default answer for most vision tasks and the thing a reviewer would ask about if it were missing.
• A custom CNN — a compact convolutional network designed from scratch for 48×48 grayscale input, sized deliberately to the data rather than to the problem's reputation.

Making that comparison mean anything required holding everything else still: the same data splits, the same augmentation, the same evaluation on a held-out test set touched only at the end. The pretrained backbones expect 224×224 three-channel input, so adapting 48×48 grayscale to them is itself a decision that shapes the result — worth being explicit about, because it's where a lot of transfer-learning comparisons quietly go wrong.

Image augmentation carried more weight here than usual. With a small dataset and a model training from scratch, augmentation is what stands between a compact network and immediate overfitting. Everything runs in TensorFlow/Keras, from the data pipeline through augmentation and training to final evaluation, reproducible end to end from the notebook.

03The results

Four architectures, one held-out test set. The custom CNN reached 82% test accuracy — 16 points clear of the best pretrained backbone:

• Custom CNN — 82%
• VGG16 — 66% · best of the transfer models
• EfficientNet — 59%
• ResNet — 54%

That isn't a narrow win. Every ImageNet-pretrained backbone finished well behind a network designed for the data, and the explanation is domain mismatch rather than model quality. ImageNet features are built from millions of full-color, high-resolution photographs of objects in scenes; the early layers encode color gradients and textures that do not exist in a 48×48 grayscale face. The pretrained models arrive carrying representational machinery for a problem they aren't being asked to solve.

The ordering among them says the same thing. VGG16, the shallowest, did best; ResNet, the deepest, did worst — a twelve-point spread running in the opposite direction to model capacity. On data this small and this far from ImageNet, depth was a liability.

The compact model also trained faster and is far cheaper to serve. On a task where the accuracy ceiling is set by human label agreement rather than by model capacity, that trade-off is the whole ballgame — the kind that matters when a model has to ship, not just score.

04What I learned

Transfer learning is a tool, not a law. Reaching for the largest pretrained backbone is the reflex, and on specialized, low-resolution data it is frequently the wrong one. The useful question isn't "is this model powerful" but "was it trained on anything resembling my input" — and here the honest answer was no. Measuring beat assuming, and it beat it by enough to change the conclusion.

Capacity has to be earned against the data you actually have. A model with more parameters than the dataset can constrain will find patterns that aren't there. Matching the architecture to 2,304 grayscale pixels wasn't a compromise forced by compute — it was the thing that made the model generalize. The ordering among the pretrained models makes the same point from the other direction: VGG16 beat ResNet by twelve points, and the difference between them is mostly depth.

The pipeline is most of the work. Clean data handling and augmentation moved accuracy more than architecture tweaks did — which is exactly what four years of production data engineering would predict, and a useful corrective to how much attention architecture gets relative to the plumbing underneath it.