Portland and the federal government

Learn about our sanctuary city status, efforts to block federal overreach: Portland.gov/Federal

Playbook

Teams often start with a solution. This playbook helps you step back and focus on the problem first. This helps reduce assumptions and risk.

A graphic illustrates the cycle of three phases: define, discover, and deliver.View full size image

Measure an entire system

City services should be easy to find, understand and use. This play helps you measure overall usability for a system or service. It uses two short survey questions to measure usefulness and ease of use. The result is a simple score you can track over time.

Use this when you want a high-level "usability health" check for an entire experience. Do not use it to measure a single task or user flow.

The outcome is a baseline score you can compare against as you make meaningful improvements.

Setup takes about one afternoon. Response time depends on how quickly you can reach participants. Plan to rerun it periodically over at least six months.

What it is

This play is a short survey with two questions:

  • One measures whether the service meets people's needs.
  • One measures how easy it is to use.

This version is adapted from UMUX-Lite, a leading industry standard, and customized for City use. It uses a 4-point scale and a simple average score. Scores should be compared within your own project over time. They are not meant to be compared to outside industry benchmarks. 

Two multiple choice questions and a submit button illustrate one way to measure.View full size image

Each participant answers both questions. You average their two scores. Then you average all participant scores together. The final number is your usability score.

This gives you a high-level view of usability health. It does not diagnose specific problems.

Use this when you want to measure an entire system or service. Do not use it when you want to measure a specific task. For that, use the measure parts of an experience play. 

How to use it

Who you'll need

  • One person responsible for this work 
  • Participants who have used the service 

What you'll need

  • A digital survey tool
  • A spreadsheet to calculate averages 

Steps

  1. Create the survey.

    Add these two questions. Keep the wording and scale consistent each time you run the survey.

    • Question 1: Did this service meet your needs?
      • 1 = Not at all
      • 2 = A little
      • 3 = Mostly
      • 4 = Completely
    • Question 2: How difficult or easy was it to use this service?
      • 1 = Very difficult
      • 2 = Somewhat difficult
      • 3 = Somewhat easy
      • 4 = Very easy 

  2. Send the survey.

    In the first session, focus on volume, not perfect wording. Discuss and capture ...

    • Send the survey (ex. Email with link, embedded website form, etc.) to people who recently used the service.
    • Aim for 30 to 50 responses when possible. More responses increase confidence in the result.
    • If access is limited, collect as many responses as you can and document the sample size. See the plan for learning play for guidance on choosing a sample size. 
  3. Calculate the score.
    • For each participant, average their two answers.
    • After you have all participant averages, average those together.
    • The result is your usability score on a 1 to 4 scale. This is your baseline. 

  4. Interpret the score.

    Use the score as a directional signal. These ranges are guidelines. Look at trends over time rather than reacting to a single score. 

    A table with a header row, five rows of data, and four columns, showing what an average of the data might look like.View full size image
    • 3.5–4.0: Strong signal that most users feel the service works well and is easy to use.
    • 3.0–3.49: Generally positive, but friction likely exists. Worth investigating.
    • 2.5–2.99: Clear signal that many users are struggling in some way.
    • Below 2.5: Strong signal that significant usability or effectiveness issues exist. 
  5. Rerun after meaningful changes.
    • Rerun the survey after major updates that affect the overall experience.
    • Do not rerun after minor tweaks. This measures the whole system, not small adjustments.
    • Track scores over time to see whether usability is improving. 

What to produce

  • A digital survey
  • A spreadsheet with calculated averages
  • A baseline usability score 

What decisions you'll make

Always remember this is a broad measure. It does not explain why users are struggling. Use this score to decide:

  • Whether usability needs focused improvement
  • Whether recent changes improved the overall experience
  • When to prioritize deeper research 

You're done when

  • You have collected responses and calculated a baseline score.
  • You have documented your sample size and date of measurement. 

Resources

  • UMUX-Lite survey form (example) 
    A sample form that shows how to write UMUX-Lite questions using a rating scale. Edit it to fit your project. (UMUX-Lite = a short survey that measures how usable something is.)
  • Score calculator (Excel document)
    An example spreadsheet that shows how to calculate UMUX-Lite scores. Use it as a guide when you set up your own sheet.  
Back Next page

See something we could improve on this page? Give website feedback.

Back to top