Survey Design and Analysis Plan

A survey instrument that will not lie to you: screener, ordered question set with the right scales, the leading questions removed and why, the sample size you actually need, and the analysis plan written before you collect a single response.

A Markdown survey design containing: (1) a screener of two to four questions with disqualifying answers marked; (2) ten to fifteen ordered questions, each with its scale type and the reason for it; (3) one or two open-ended questions placed deliberately; (4) a sample-size range with reasoning; (5) an analysis plan written before collection, stating the comparisons and which result supports which decision; (6) an 'excluded questions' section of four or five questions with the bias each would have introduced.

# Survey design — mobile app or mobile notifications

**Before anything else:** the question as posed cannot be answered by asking people which they want. Ask a user whether they want a mobile app and a large share will say yes, because it costs them nothing to say so. This instrument is built to find out what they currently *do* on a phone and what they currently *miss*, and to let the app-versus-notifications conclusion fall out of behaviour rather than preference.

## Screener
1. Have you used [product] in the last 30 days? — Yes / No. **Disqualify: No.**
2. Which best describes your role? — [list] / Other. *Not disqualifying; used for cross-tabs.*
3. Do you carry a work-enabled smartphone? — Yes / No / Prefer not to say. **Disqualify: No** — they cannot use either option.

## Questions

**1. In the last two weeks, how often have you tried to use [product] from a phone?**
Frequency scale: Not at all / Once / 2-5 times / 6-10 times / More than 10 times.
_Behavioural frequency, bounded recall period. This is the single most important question in the survey and it goes first, before any mention of an app plants the idea._

**2. The last time you tried, what were you trying to do?**
Multiple choice with an 'other' free-text: check a status / respond to something / show someone / find a piece of information / start a piece of work / something else.
_Distinguishes consumption from creation, which is the whole app-versus-notifications question in one item._

**3. What happened?**
Single choice: I did it on the phone / I did it later on a computer / Someone else did it / It did not get done / Does not apply.
_The failure mode matters more than the attempt. 'Did it later on a computer' is a very different signal from 'it did not get done'._

**4. In the last two weeks, how often did you find out about something in [product] later than you would have liked?**
Frequency scale as question 1.
_This is the notifications hypothesis, asked without mentioning notifications._

**5. When that happened, how did you eventually find out?**
Multiple choice: email / a colleague told me / I checked and saw it / Slack or Teams / I did not find out until it was a problem / Does not apply.
_Reveals whether an existing channel already covers it, which would make both options unnecessary._

**6. How much of your working week do you spend away from a computer?**
Scale: None / Under 10% / 10-25% / 25-50% / More than 50%.
_The strongest available proxy for whether a mobile app is structurally necessary for this respondent._

**7. Do you currently get any notifications from [product] anywhere?**
Multiple choice: email / Slack or Teams / browser / none / not sure.

**8. If you get them, how do you handle them?**
Single choice: read and act immediately / read and act later / skim / ignore / turned them off.
_If a meaningful group has turned notifications off, the notifications hypothesis is in trouble and this is the item that surfaces it._

**9. Which of these have you done in the last month on a phone, for work generally, not our product?**
Multi-select: approved something / replied to a work message / reviewed a document / made an update in a work tool / none of these.
_Calibrates against their general mobile-work behaviour. Someone who never approves anything on a phone will not start with you._

**10. Think about the last time [product] would have been useful on a phone. What were you trying to get done?**
Open text, optional.
_The single open question, placed after nine short items and before the demographics, where response quality is highest._

**11. How many people at your organisation use [product]?**
Bands: 1 / 2-10 / 11-50 / 51-200 / More than 200 / Not sure.

**12. Which best describes your organisation's industry?**
[list] / Other / Prefer not to say.

## Sample size
**Aim for 300-400 completed responses from roughly 4,000 active users.** That is enough to compare two or three subgroups (by role, and by time away from a computer) with differences you can act on. Below about 150 you can read the whole-sample direction only, and any subgroup claim will be noise.

This is a rule-of-thumb range, not a power calculation. If this decision is worth two quarters of engineering, it is worth twenty minutes of a statistician's time to check.

## Analysis plan — commit to this before you send it

**Primary comparison:** question 1 (attempts to use on a phone) crossed with question 3 (what happened).
- If most attempts end in 'I did it later on a computer', the need is deferral-tolerant and **notifications win**.
- If a meaningful share end in 'it did not get done', there is real unmet demand for doing work on a phone and the **app case strengthens**.

**Secondary comparison:** question 2 (what they were trying to do) split by consumption versus creation. If checking status and responding dominate, notifications plus a mobile-readable view covers the need. If starting work appears often, that is an app.

**Segmentation:** question 6 (time away from a computer). If mobile demand concentrates entirely in the above-25% group, size that group before building for it — it may be a real segment or it may be 8% of your users.

**Kill criterion:** if question 1 shows most respondents have not attempted phone access at all in two weeks, neither project is justified by this data, and the honest report says so.

## Questions deliberately excluded

- **"Would you use a mobile app if we built one?"** — everyone says yes. It measures agreeableness and it would be quoted in the roadmap deck for the next two years.
- **"How important is mobile access to you? (1-5)"** — importance ratings without a trade-off are uniformly high and cannot discriminate between the two options.
- **"Would you pay more for a mobile app?"** — stated willingness to pay is unreliable in surveys and worse in B2B, where the respondent is usually not the payer. Someone will ask for this question. The answer is no.
- **"Rank these upcoming features in order of preference."** — produces a ranking of what sounds good, invites lobbying, and does not tell you what anyone would use.
- **"Do you find the current mobile experience frustrating?"** — leading, double-barrelled with an assumption, and answerable only in the direction the question suggests.