Zikr Apps · Engineering notes
From flaky to flawless: making our app tests trustworthy
We set out to run our Misbaha app through 150 automated tests and trust the result. It started at a frustrating 84%. This is the story of why automated tests lie, the dead ends we hit, and how we reached a clean 100% — with before-and-after pictures for every problem.
- Automated runs
- 150
- Final pass rate
- 100%
- Distinct issues fixed
- 13
- Worst point
- 84%
Last updated June 2026
Meet Misbaha
Misbaha is a digital prayer-bead (tasbeeh) and supplications app: count your dhikr, keep daily streaks, set goals, and explore a library of authentic duas — all in a calm, distraction-free space. Here is a 75-second look at the app.
Get Misbaha
Misbaha is live on the Apple App Store. The Android build is available as a direct download while the Google Play listing is still in review.
The Google Play Store
listing is not live yet — until it is, install the Android version using the
.apk above (you may need to allow installs from your browser).
Open source
The code is on GitHub
Misbaha is open source. The full app — every screen, the test journeys, and the reliability suite described on this page — lives in one public repository. Browse the code, file an issue, or follow along with how it is built.
How Misbaha is built
A modern React Native stack, shipped to iOS and Android from a single codebase.
App framework
- Expo SDK 54
- React Native 0.81
- React 19
Language
- TypeScript
Navigation
- Expo Router 6 (file-based)
State & storage
- Zustand
- AsyncStorage
UI & motion
- React Native Reanimated 4
- react-native-svg
- Expo Haptics
QA automation
- DevLoop / QualGent
- iOS Simulator
- WebDriverAgent
What this is about
Before we ship an update to Misbaha (a digital prayer-bead and supplications app), a fleet of automated tests "uses" the app the way a person would: opening it, tapping around, counting, creating goals, searching. If a test fails, it should mean the app is broken. The trouble is that tests can also fail for reasons that have nothing to do with the app — and telling those apart is most of the work.
To measure how trustworthy our app really is, we run each of 5 core journeys 30 times — 150 runs in total. A reliable app should pass all of them, every time. Ours kept hovering between 84% and 95%. Below is everything that stood between us and a clean 100%, grouped into a simple category so you can see the pattern: most failures were never the app at all.
Where we started
84%
126 of 150 runs
Where we landed
100%
150 of 150 runs
How we test
Reliability is not a single number you check once — it is something you have to measure carefully and repeatedly. Here is the approach in plain terms.
- Five everyday journeys
- We picked the five things people actually do in Misbaha and scripted each one end to end, the way a real person would tap through it.
- Thirty times each
- We run every journey 30 times — 150 runs in total. Flakiness hides from a single run: a glitch that strikes 1 time in 20 is invisible unless you repeat the test enough to catch it.
- A clean slate every time
- Before each run the app is reset to a known starting state and we wait until it is genuinely ready, so one test can never contaminate the next.
- One phone at a time
- After discovering the engine was tripping over itself, we run on a single simulated phone, one journey after another, for dependable results.
- All-or-nothing scoring
- A run only counts as a pass if every required step in the journey succeeds. Anything less is a fail.
- Measure, fix, measure again
- We measure reliability first (the "baseline"), fix the causes of every failure, then measure again (the "hardened" run) to prove the improvement.
The five journeys
These are the five things we make sure work, every single time, before any update reaches our users.
- 1
Open the app
Cold-start Misbaha and land on a ready Today screen with no pop-ups in the way.
- 2
Count a dhikr
Open the Tasbeeh tab, choose a supplication, count it, and confirm the tally goes up.
- 3
Search for a supplication
Search the duas by name and open a matching result.
- 4
Start a surprise goal
Begin a randomly chosen multi-day goal and confirm it appears in the active goals list.
- 5
Leave and come back
Count, switch away from the app, return, and confirm the count was preserved.
Results by journey
Each journey ran 30 times before our fixes (baseline) and 30 times after (hardened). The pattern is clear: every journey reached a perfect score.
| Journey | What it checks | Baseline | Hardened |
|---|---|---|---|
| Open the app | Stable launch | 27/30 (90%) | 30/30 (100%) |
| Count a dhikr | Counting works | 21/30 (70%) | 30/30 (100%) |
| Search for a supplication | Search + open | 25/30 (83%) | 30/30 (100%) |
| Start a surprise goal | Goal created | 24/30 (80%) | 30/30 (100%) |
| Leave and come back | State persists | 29/30 (97%) | 30/30 (100%) |
| All journeys | 126/150 (84%) | 150/150 (100%) | |
Baseline = the run before the final round of fixes (84.6% overall). Hardened = the run after all 13 issues were resolved (100% overall, with no connection errors).
The reliability scorecard
Pass rate is only part of the story. These are the measures QA engineers use to judge whether a test suite can be trusted — before our fixes, and after.
| Measure | What it means | Baseline | Hardened |
|---|---|---|---|
| Success rate | How often a journey passes | 84.6% | 100% |
| Flake rate | Runs that disagreed with the usual result | 15.4% | 0% |
| Reproducibility | Journeys that gave the same verdict every time | 0 of 5 | 5 of 5 |
| False alarms | Failures where the app was actually fine | all 23 failures | 0 |
| When failures struck | How far into a journey it broke | at the first step | n/a — none failed |
| Diagnosed failures | Failures we could explain and categorise | 100% | 100% |
The most telling row is "false alarms": every one of the baseline's 23 failures was the test giving up too early, not a real problem in the app.
Every issue, before and after
Each card shows the problem in plain terms, then a picture of what was happening before the fix and what happens after — followed by the technical detail for engineers.
1 Opening the wrong app
Test config
Think of every app on a phone as having a unique badge number. Our test handed out the wrong badge number, so it kept knocking on the wrong door.
Before
After
The fix: Corrected the app's badge number (its bundle identifier) everywhere the test referred to it.
› Tech details
The harness's APP_BUNDLE constant was stale. Every apply_routine / mobile_launch_app call targets a bundle id, and it must match the installed binary. Set it to com.zikrapps.misbaha across the harness and routine definitions.
2 Too many phones at once
Parallel vs. single-thread
Running many simulated phones at the same time is like opening too many programs on an old laptop — everything slows to a crawl and things start failing for no real reason.
Before
After
The fix: Reduced from 5 simultaneous phones to a single phone running tests one after another.
› Tech details
Five concurrent simulators saturated the CPU on a 16 GB M4 Air; XCUITest steps blew past postcondition timeouts. Each simulator also runs its own WebDriverAgent, so per-device overhead — not test logic — was the bottleneck. Dropped to 2, then to single-sim sequential.
3 The app built with the wrong ID
Build / tooling
Even after fixing the test, the app itself had been manufactured with the wrong badge number printed on it. Both sides have to match.
Before
After
The fix: Set the correct identifier in the app build settings, then rebuilt and reinstalled it.
› Tech details
PRODUCT_BUNDLE_IDENTIFIER in project.pbxproj was com.misbaha.app for both Debug and Release. Rebuilt with com.zikrapps.misbaha so the installed id matches the routine target.
4 A pop-up kept blocking everything
Test flow
To save time, tests used shortcuts to jump between screens. But each shortcut made the phone ask "Are you sure you want to open Misbaha?" — and a question nobody answered stopped the test cold.
Before
After
The fix: Switched navigation to tapping the app’s real buttons, and taught the test to automatically confirm any pop-up that still appears.
› Tech details
Custom-scheme deep links (misbaha://test/go/{tab}) trigger the iOS "Open in app?" confirmation. Replaced open_url navigation with tap_element on tab testIDs, and added a confirm_scheme_dialog helper that taps "Open" when a prompt is detected.
5 The app crashed when tests moved too fast
Real app bug
A person taking a fraction of a second to react never notices, but a robot is instant. It told the app to switch screens before the app had even finished opening, and the app fell over.
Before
After
The fix: Made the app defer the screen change by one frame, so it only navigates once it is fully mounted and ready.
› Tech details
expo-router router.replace() ran inside a test route useEffect before the Root Layout mounted → "Attempted to navigate before mounting the Root Layout". Deferred the call with requestAnimationFrame in app/test/reset.tsx and app/test/go/[tab].tsx.
6 Errors were happening silently
Real app bug
The app had a safety net that caught crashes so users would not see them — but it threw the details away, leaving us blind to what actually broke.
Before
After
The fix: Added error logging to the safety net so every caught error is recorded with its cause.
› Tech details
The ErrorBoundary implemented getDerivedStateFromError for fallback UI but had no componentDidCatch, so stacks were swallowed. Added componentDidCatch logging to surface the error and component stack.
7 Phones got "stuck" and unavailable
Infrastructure
Each test "reserves" a phone, like checking out a library book. When a test crashed hard, it never returned the book — so the next test was told everything was checked out.
Before
After
The fix: Cleared the stuck reservations and ensured runs always release their phone, even when they fail.
› Tech details
Force-killed harness processes left qg_acquire_device leases dangling (no finally ran), so the bridge returned device-busy from another "external session". Guaranteed release in a finally block; cleared orphaned leases with a bridge restart.
8 Stuck on "Downloading 100%"
Build / tooling
The test version of the app downloaded its code live from a developer server every launch. When that server hiccuped, the app froze on a loading bar.
Before
After
The fix: Built a self-contained release version with all code bundled inside, removing the live download.
› Tech details
The dev client streamed its JS bundle from Metro; a stall left the app on "Downloading 100%". Switched to a Release build with the Hermes bundle embedded (keeping EXPO_PUBLIC_E2E=1), eliminating the Metro runtime dependency.
9 Looking for a button that had moved
Test maintenance
The counting test was written for an older version of the screen. The app had since been redesigned, so the test reached for things that were no longer there.
Before
After
The fix: Rewrote the test against the current screen and used a steady press-and-hold gesture instead of fragile rapid taps.
› Tech details
expand_dua_and_count referenced a removed category and timing-fragile double taps. Rebuilt as v8: open "Quranic Duas", then a zero-displacement swipe_coordinates (~3.2 s long-press) for a deterministic count increment.
10 Tapping before the screen was ready
Test timing
The "new goal" test moved to a screen and immediately tapped a button there — but under load, the screen took an extra moment to draw, so the tap missed.
Before
After
The fix: Added a "wait until the screen has changed" check before each tap on that flow.
› Tech details
start_surprise_goal tapped surprise-goal-button before the Goals screen rendered, failing 7/30 under load. Hardened as v4 with wait_for: screen_changed (6 s timeout) on the navigation steps.
11 Starting before the app woke up
Test reset logic
Between tests we restart the app. We used to wait a fixed number of seconds — but on a busy laptop that guess was too short, so tests began while only the splash logo was showing.
Before
After
The fix: Replaced the fixed wait with an active poll that proceeds only once real home-screen content is on screen.
› Tech details
hard_reset used fixed sleeps; under load the launch splash outlasted them and the first tap hit a bare splash ("Visible: Misbaha"). Added wait_until_ready, which polls mobile_observe_screen for home markers before any routine runs.
12 The robot kept losing connection
Infrastructure
The automation engine occasionally dropped its connection mid-test. A momentary network blip is not a real product problem, but it was being counted as a test failure.
Before
After
The fix: Added automatic retries with a short backoff and a recovery step, so transient blips no longer fail a test.
› Tech details
Transient bridge disconnects ("All connection attempts failed" / "Server disconnected") surfaced as execution_error. apply_with_retry catches RuntimeError and recoverable traces, backs off (min(4·n, 15) s), and runs recover_to_today between attempts; real failures still return immediately.
13 Two robots fighting over one phone
Infrastructure (root cause)
This was the real villain. The automation engine was quietly starting two identical control programs on the same phone. Both tried to use the same single "doorway" (a network port), collided, and crashed — taking the whole test down with them. Restarting one phone helped briefly, then it came back.
Before
After
The fix: Ran a single phone sequentially and restarted the automation engine to clear the duplicate control programs and stuck reservations.
› Tech details
The crash report showed WebDriverAgent aborting in -[FBWebServer startHTTPServer] (FBWebServer.m:124) — a GCDWebServer port-bind failure caused by duplicate "xcodebuild test … WebDriverAgent" launchers targeting the same simulator. The second instance could not bind the WDA port and aborted, so every apply_routine returned "All connection attempts failed". Fixed by single-sim sequential execution plus a QualGent Desktop restart to clear duplicate runners and stale leases.
Sorting failures by type
Good QA work does not just count failures — it sorts them into standard types, so you know whether to fix the app, the test, or the machine. Here is where our 13 issues landed, including two types we designed out entirely.
App bug
Genuine defects in the app, found by the tests and fixed in the app’s code.
Test design bug
The test itself was wrong — pointing at the wrong thing or making a bad assumption.
Agent / timing error
The test acted before the app had finished drawing. This caused all 23 of the baseline run failures.
Environment / infrastructure
Device, build, or test-engine plumbing broke — including the root-cause crash.
State drift
Prevented by design: every run starts from a clean, reset state.
Weak check (oracle)
Prevented by design: we check real numbers (the count went up by 3), not just that "some text exists".
Unknown
Every single failure was explained — nothing left unclassified.
Classification coverage: 100% — every failure was explained, none left as "unknown".
How to read the categories
- Real app bug
- A genuine defect in the app, fixed in the app’s code.
- Infrastructure
- A problem in the testing engine or device plumbing, not the app.
- Parallel vs. single-thread
- Caused by running too much at once on one machine.
- Build / tooling
- How the app was packaged for testing.
- Test config / maintenance
- The test scripts pointed at the wrong thing or had gone stale.
- Test timing / flow
- The test acted faster than the app could respond.
The exact setup
For anyone who wants to reproduce this, here is precisely what was tested and how.
- App
- Misbaha 1.2.0 (build 3)
- Platform
- iOS 26.3 — iPhone 17 Pro Simulator
- Test engine
- DevLoop 0.0.96 (QualGent)
- Runs per batch
- 5 journeys × 30 = 150
- Execution
- One simulator, sequential
- Clean slate
- Reset deep link + readiness wait before each run
Honest limitations
A trustworthy report says what it cannot yet claim. These are the open trade-offs.
- Running one phone at a time is slower than running many — that was the trade we made for dependable results until the test engine stops launching duplicate controllers.
- The surprise goal is random on purpose, so we check that a goal was created, not which one.
- Because we test a packaged build, code changes need a rebuild before the tests exercise them.
- These numbers come from one before batch and one after batch; more batches would tighten the confidence further.
The takeaway
Of the 13 problems we fixed, only a couple were genuine app bugs. The rest were the testing setup itself: machines overloaded, tools misconfigured, and an automation engine quietly tripping over its own feet. That is the real lesson of reliability work — before you can trust what a test says about your app, you have to make the test itself trustworthy.
Once we did, the picture was unambiguous: 150 out of 150 runs passed, with no connection errors at all.