Preparing Video Frames for a Computer Vision Dataset

The mistake that produces 10,000 useless images

The obvious approach to turning video into training data is to extract every frame. It feels thorough. It is close to the worst thing you can do.

Consecutive frames of 30 fps video are 33 milliseconds apart. Unless the camera or subject is moving fast, frame 1,001 is nearly identical to frame 1,000 — same pose, same lighting, same background, a few pixels of difference. A dataset of 10,000 such frames does not contain 10,000 examples. It contains perhaps 300 distinct situations, each repeated thirty times.

What that does to training is specific and damaging. Annotation cost scales with image count, so you pay thirty times over for the same information. Training time scales too. And most importantly, if those near-duplicates end up on both sides of your train/validation split, the model has effectively seen the validation set — you get a validation score that means nothing.

So the first decision is not a tool setting. It is: how much does the content have to change before a frame counts as a new example?

Choosing the sampling interval

Derive it from the motion in the scene, not from the source frame rate. Some working starting points:

• Static camera, slow subject (surveillance, a plant growing, a queue) — 1 frame every 2–10 seconds. Anything denser is redundant.
• Static camera, active subject (a person working at a bench, traffic) — 1 frame every 0.5–2 seconds.
• Moving camera (handheld, drone, dashcam) — 2–5 frames per second. The background changes continuously, so frames stay distinct.
• Fast action where the pose itself is the label (sport, gesture recognition) — 5–15 frames per second, but only across the segments that contain the action.

The way to settle it: extract a one-minute test segment at a few intervals and look at the results side by side. If you can page through them and each image tells you something new, the interval is right. If it feels like a flipbook, widen it.

In the extractor, "Time Interval" mode takes the gap in seconds directly, and the frame count appears as you type — so the number above is visible before you commit rather than discovered afterwards.

Extraction settings that matter downstream

Format: PNG or JPEG

The honest answer is that it usually does not matter as much as people expect, but the cases where it does are worth knowing.

Use PNG when the task is sensitive to fine detail or compression artefacts — defect detection on surfaces, OCR-adjacent work, medical or scientific imaging, anything where you might later measure pixel values. Use JPEG at 90–95% for general object detection and classification, where models are robust to mild compression and the storage difference is five to tenfold.

One consistency rule regardless of choice: do not mix. A dataset that is half PNG and half JPEG at varying quality gives the model a spurious signal, and models are very good at finding spurious signals.

Resolution

Extract at source resolution and downscale in your preprocessing pipeline, not at extraction time. You cannot recover detail you discarded, and a year from now you may want 512-pixel crops from footage you extracted at 224. Storage is cheaper than re-extraction, especially if the source footage is archived somewhere inconvenient.

Naming

Zero-padded sequential names, wide enough for the largest set you will ever produce. frame_00001.png, not frame_1.png. Tools that sort filenames as text will order frame_10 before frame_2, and if temporal order matters anywhere in your pipeline, that bug is subtle and hard to spot.

Include the source in the path rather than the filename, so provenance survives:

dataset/
                  raw/
                    cam01_2026-03-14/
                      frame_00001.png
                      frame_00002.png
                    cam02_2026-03-14/
                      frame_00001.png
                  splits/
                    train.txt
                    val.txt
                    test.txt

Keeping the split as text files listing paths — rather than physically copying images into train/ and val/ directories — means you can re-split without re-copying, and you can see exactly what went where.

The split rule that actually matters

This is the part worth reading twice, because it is the difference between a benchmark you can trust and one you cannot.

Split by source video, never by frame.

If you shuffle all frames and take a random 80/20 split, frames from the same second of the same video land on both sides. The validation set is then almost a copy of the training set, and your validation accuracy measures memorisation. Models trained this way routinely report 97–99% and then fail on genuinely new footage, and the failure is baffling until you look at how the split was made.

The correct approach: assign whole videos to train, validation and test. If you only have one long video, split it by time into contiguous blocks — first 70% train, next 15% validation, last 15% test — and discard a few seconds either side of each boundary so no near-duplicate pair straddles it.

Where possible, go further: hold out entire recording sessions, cameras, or locations for the test set. That measures whether the model generalises across conditions rather than across timestamps, which is nearly always the question you actually care about.

Working locally, and why it often matters here

A practical note specific to this domain: research and industrial video is frequently material that cannot casually be uploaded — patient recordings, footage of people who consented to a study and not to a cloud service, proprietary manufacturing lines, pre-publication experiments.

Browser-based extraction keeps the file on the machine; nothing is transferred, which matters when the constraint is an ethics approval or a data processing agreement rather than a preference. You can confirm this rather than take it on trust — load the page, disconnect from the network, and extract. If it still works, nothing was going anywhere. The method for verifying that takes about a minute.

Be equally clear about the limits. Frames are held in memory until download, so long sources need to be processed in time-range chunks. For a repeatable pipeline over hundreds of files, a local FFmpeg script is the right tool — the browser is for exploratory work, for one-off extractions, and for cases where you want to see the frames while deciding the interval.

A checklist before annotation starts

Annotation is the expensive step. Catching these before it begins saves the most time:

1. Page through 50 random frames. Are they visibly different from each other? If not, widen the interval and re-extract — now, not after labelling.
2. Check class balance. Uniform time sampling gives you frames in proportion to how long things were on screen, not in proportion to how much you care about them. Rare events need targeted extraction from their specific time ranges.
3. Remove the dead frames. Fades, black frames at cuts, lens flare, someone's hand over the lens. They are noise and they cost annotation time.
4. Confirm the split is by video or by time block, not by shuffled frame.
5. Record the provenance. Source file, interval, extraction date, format and quality, alongside the dataset. In six months you will need to reproduce or extend it, and reconstructing "what interval did I use" from the images is unpleasant.

None of this is difficult. It is just decided before extraction rather than discovered after annotation, which is the whole difference.