WHAT I'm WORKING ON
The challenges of Snapshot testing on CI
Last week, I've spent several days on setting up CI for RocketSim. So far, I've been working alone and didn't need much of validation. However, adding new developers to my team made it a must to ensure stability (AI agents 👀).
I've had a classic week in which I did a lot, but felt like I did nothing. Setting up CI can be a slow process, especially if you constantly think "this is it" and you'll have to wait a few minutes for CI to complete.
Along the way, I learned a few things:
Let CI record your snapshots
Niek (our frontender) told me they're doing this on the web all the time. After some research, I found out that it's quite common on iOS/macOS development as well.
Funny though—after CI recorded the snapshots, they started to fail locally due to a mismatch in machines. CI is headless, non-retina, whereas I'm running on an M3 Max. However, it did allow me to validate the failures locally (crucial for productivity) and it turned out that they actually failed for good reasons.
For context, I've generated snapshots for all types of recordings for RocketSim. This includes validating App Store Connect export resolution, recordings with bezels, and screenshots of any Simulator with device bezels. The snapshots are either the screenshot or a frame of the output video.
These things I did wrong:
- I did not respect retina scaling, resulting in different outputs on non-retina machines.
- Uneven output resolutions got corrected by AVExportSession, but only sometimes.
The latter was frustrating to find out, but it did result in several important fixes to this core library of RocketSim. Eventually, it turned out it doesn't matter whether I record on CI or locally. Both suites complete successfully now, resulting in more stability securance.