DES303 Week 7: Testing Focus Integrity With Users - Fairness, Privacy, and Trust

Introduction

In Week 6, user testing and crit feedback revealed that the central problem in Tickers was not simply whether the prototype worked, but whether users could trust the system's judgement. The feedback showed that Focus Integrity could not be treated as a simple score. If it affected arena results, proof, money, reputation, or social pressure, then the score needed to feel explainable, fair, and contestable.

This week I used the model comparison system from Week 6 as a research instrument. The goal was not to prove that the score was correct, but to test whether users understood and trusted the judgement. Instead of adding more arena features, I focused on whether the scoring layer felt fair, explainable, and acceptable to people using it.

By the end of the week, I realised that this assumption was limited. The system could collect data, replay sessions, compare models, and generate structured evidence, but more data did not automatically create better understanding. The experiment showed that Ticker was becoming better at watching the user, but not necessarily better at understanding the task.

The main learning from Week 7 is that Focus Integrity should not start from surveillance data. It has to start from task understanding.

Experience

Why I chose to investigate Focus Integrity

The first crit made the Focus Integrity problem very clear. Even though the prototype was still early, people immediately questioned the accuracy and ethics of the score. This mattered because the whole Tickers system depends on Focus Integrity being believable. If the score is wrong, the arena result becomes unfair. If the score feels hidden or invasive, users may reject the system. If the score is connected to money, reputation, proof, or social pressure, then wrong judgement has real consequences.

So I tried to make my version task-based rather than only activity-based. Instead of asking whether the user was active, Ticker should ask whether the user's observable behaviour matches the task they claimed they were doing.

What is new this week?

Week 7 overlaps with Week 6 because it continues the Focus Integrity problem, but the purpose changed. Week 6 established the trust issue. Week 7 tested whether my first technical answer to that issue was strong enough.

Experiment 2: Testing Focus Integrity With Friends

Evidence of what I actually worked on this week

To make the week more evidential, I treated Focus Integrity as a technical design experiment rather than only a conceptual reflection. I documented the work in four layers: implementation evidence, session evidence, model-comparison evidence, and peer-interpretation evidence. This mattered because the question was not only whether I could describe a scoring system, but whether I could show how the system was built, what data it produced, where it failed, and how that failure changed the direction of the project.

focus-integrity/
  collectors/
    ActiveWindowCollector.ts
    InputActivityCollector.ts
    CameraPresenceCollector.ts
    ScreenDiffCollector.ts
    ChromeActivityCollector.ts
    SystemStateCollector.ts
  pipeline/
    FeatureExtractor.ts
    CertaintyGate.ts
    AIJudge.ts
  scoring/
    TriScoreEngine.ts
    SessionStateMachine.ts
  research/
    ResearchTickReplay.ts
    modelComparison.ts

[tracker] session started: DES303 Week 7 blog
[collector] active_window=Chrome title="DES303 brief"
[collector] input=low idle=38s screen_diff=0.04
[ai-judge] verdict=partial confidence=0.61
[triscore] integrity=74 engagement=42 confidence=58
[research] saved tick #034 to replay buffer
[replay] models=M0, MA, MS, MB, MAI, CPRES, MPRES

TriScore model — building my first live Focus Integrity architecture

The main research instrument I used this week was the TriScore model. This was my first working version of Focus Integrity as a live system rather than only a product idea. The purpose of the model was to replay and inspect focus sessions in small repeated “ticks”, collect behavioural evidence from the desktop, and test whether the user's current activity still matched the task they had committed to.

This mattered because the Week 6 critique showed that Focus Integrity could not remain as a vague score. If Ticker was going to use the score to affect streaks, arena results, proof, or accountability, then I needed to understand how the judgement was being made. Building the TriScore model was therefore not just technical development. It was a design research experiment into whether focus can be judged fairly through observable signals.

The live system ran in this loop:

Collectors → Feature Extractor → Certainty Gate / AI Judge → TriScore Engine → State Machine → Intervention UI

This structure helped me test what would happen if Ticker judged focus continuously during a session. It also made the weakness of the approach more visible. The system could collect more evidence, but it still struggled to know what the evidence meant.

FIGURE 4 · LIVE FOCUS INTEGRITY TRACKER

The system collected raw signals from the desktop every few seconds through several collectors:

This collector table was useful because it made the system's assumptions visible. Each collector translated part of the user's behaviour into data, but none of them could fully explain intention. For example, the ActiveWindowCollector could tell that Chrome was open, but not whether I was researching, watching a lecture, checking documentation, or drifting away from the task. The InputActivityCollector could detect low typing, but low typing might mean reading, planning, thinking, or being distracted. This showed me that adding collectors improved visibility, but did not automatically improve understanding.

FIGURE 5 · RAW TICK EVIDENCE

timestampactive appinput statecamera statescreen diffAI verdictintegrity / engagement / confidencehuman label

{
  "tick": 34,
  "timestamp": "2026-04-18T14:22:40+12:00",
  "declaredTask": "write DES303 Week 7 blog",
  "activeApp": "Google Chrome",
  "windowTitle": "DES303 Week 7 2026 - Canvas",
  "input": { "typing": "low", "idleSeconds": 38 },
  "presence": { "camera": "face_not_detected" },
  "screen": { "diffRatio": 0.04, "state": "static_reading" },
  "aiVerdict": { "label": "drift_risk", "confidence": 0.61 },
  "triScore": { "integrity": 74, "engagement": 42, "confidence": 58 },
  "humanLabel": "aligned / partial"
}

The Feature Extractor then converted this raw telemetry into higher-level features using a rolling buffer of recent ticks — approximately a few minutes of history — and calculated things like app switching rate, dwell time, focus continuity, idle seconds, doom scrolling, typing bursts, face stability, downward glances, screen-diff ratios, and task keyword matching.

How the live TriScore model used AI

I added a certainty gate before the AI Judge so I wouldn't call AI on every tick. If local evidence was clear, the system decided locally. If evidence was unclear, the backend formatted an evidence bundle and Claude returned a structured judgement — on-task yes/no, confidence, and a short reason. That verdict fed back into the TriScore Engine and could move the user through warning, recovery, or failed states.

At this stage the system felt promising — it could respond while the user was working. But it created a serious issue: if the judgement was wrong, the system could interrupt the user unfairly.

One ambiguous test case: when the system could watch but not understand

This ambiguous case became the most important evidence of the week. It showed that the problem was not only model accuracy. The problem was meaning. The same signal could be interpreted in opposite ways depending on the task. This meant the system could be technically detailed and still unfair.

This was the point where the experiment failed productively. The failure was useful because it changed the direction of the project. I no longer wanted to keep adding more surveillance signals. I needed to redesign the system so that it understood the work before judging the behaviour.

Research Tick / Replay model — the investigation system

I therefore created a separate Research Tick / Replay system. This was not designed to control the user live. It was designed to help me investigate which data was truly meaningful.

Because Week 6 had already introduced the score-check dashboard, I used Week 7 to push the dashboard further as a research tool. The main change was not simply adding more models, but using the replay system to ask why each model still failed in ambiguous task situations.

This changed Focus Integrity from a single-score problem into a comparison problem. I was no longer asking whether one model was “right”. I was asking which evidence types improved judgement, which were too invasive, and which were too expensive or unreliable to use.

FIGURE 7 · RESEARCH TICK MODEL COMPARISON SCREEN

FIGURE 9 · EXPLORATORY MODEL-LABEL AGREEMENT

How I tried to correct the data

My idea was that users could correct segments when the system was wrong — labelling them as aligned, partial, unclear, or not aligned. These corrections would become ground truth so the system could learn each user's focus pattern over time. This connects to data labelling in machine learning, where models need labelled examples to interpret raw data correctly.

FIGURE 10 · USER CORRECTION UI

Friend testing: fairness, privacy, and trust

After building the Research Tick replay system and user correction overlay, I ran a small formative test with four friends / classmates. This was not intended to prove that the model was accurate. The aim was to test whether people could understand what the Focus Integrity score was trying to communicate, which evidence signals felt acceptable, and where the judgement felt unfair or invasive.

I showed users the live tracker, replay dashboard, and ambiguous segments from short focus sessions. I then asked them to explain what they thought the score meant, which evidence they trusted, which evidence felt too invasive, and what they would want to happen if the system judged them incorrectly. This was a small qualitative check. NN/g argues that small formative usability tests are useful for finding design problems and supporting iteration, while larger samples are needed for quantitative claims (Nielsen, 2000, 2012).

EXPERIMENT 2 VISUAL · PARTICIPANT SESSION

EXPERIMENT 2 VISUAL · PLANNED DISTRACTION MOMENT

The user test confirmed that the problem was not only technical accuracy. Even if the model produced a score, users still needed to understand why the score happened and what they could do when it was wrong. This supported my Week 7 finding that tick-based evidence is not enough by itself. The next version needs task context, uncertainty, and clarification before judgement. Combining implementation evidence, replay evidence, and peer interpretation also helped me triangulate the finding instead of relying on one evidence source (Whitenton, 2021).

Reflection on Action

My main learning

The biggest learning this week was that my Focus Integrity research method was too focused on accuracy and not enough on meaning. At the start of the week, I thought the main problem was that the score needed more evidence. My logic was:

more ticks → more collectors → more model comparison → more user correction → more reliable Focus Integrity

After building the TriScore model and the Research Tick / Replay system, I realised that this assumption was limited. The system could become more technically detailed, but that did not mean it became more fair or more useful.

The issue was not only whether the model had enough data. The issue was whether the model understood what the user was trying to do. Chrome could be distraction or research. A static screen could be inactivity or deep reading. Low typing could be avoidance or thinking. YouTube could be entertainment or learning. Without task context, the same behaviour could produce the wrong judgement.

This changed my understanding of Focus Integrity. I no longer saw it as a simple detection problem. I began to see it as a task-alignment problem. The system should not ask only, “Is the user active?” or “Does this behaviour look focused?” It should ask, “Does this behaviour make sense for the task the user committed to?”

This shift was important because it changed the direction of the project. Before Week 7, I was trying to make Ticker a better focus detector. After Week 7, I started to question whether detection was the right starting point at all. The experiment showed that surveillance can create confidence without creating understanding. A system may look more intelligent because it collects more data, but if it does not understand the task, it can still make unfair or unhelpful judgements.

Why this failed as a design direction

Ticker was not made only so that AI could check whether a user is focused. The larger purpose was to help the user focus further. The tick-based method did not do that well. It mainly monitored the user and reacted after it suspected drift. It did not properly structure the work before the session. It did not understand why the user chose a task, or help them decide what to do next.

My broader project tension is about AI increasing efficiency while surveillance becomes normalised. But this version mostly increased surveillance without enough improvement in efficiency. The more believable future is a system that becomes useful enough that people accept being watched.

My positionality as a designer

This week also showed me something about my own design pattern. I am comfortable building systems — infrastructure, data models, backend logic, scoring engines, full product flows. This is a strength. It is also a weakness because I sometimes treat design uncertainty as an engineering problem. I responded to the trust problem by building more architecture: more collectors, more models, more replay systems. The deeper I went into tick-based detection, the more I realised the issue was not missing logic. It was missing context.

This connects back to my wider design pattern. Because I am comfortable building full systems, I often respond to uncertainty by adding infrastructure. In Week 7, that strength became a blind spot. I made the system more complex before fully questioning whether behavioural detection was the right starting point.

Theory

Why behaviour data alone was not enough

The Week 7 experiment needed research grounding because Focus Integrity sits between productivity tracking, artificial intelligence, and surveillance. At first, I treated Focus Integrity as a detection problem: if the system collected enough behavioural signals, it could decide whether the user was focused or distracted. The experiment showed that this was too simple.

Context-aware computing

Dey (2001) defines context as information that helps characterise the situation of a person, place, or object. For Ticker, this means active app, window title, typing activity, and screen movement are not meaningful by themselves. They only become meaningful when connected to the user's actual task.

Dourish (2004) also reframes context as something produced through activity and interpretation, not a fixed background variable. This directly matched my ambiguous test cases. Chrome, YouTube, a static screen, or low typing cannot automatically be classified as distraction because the same signal can mean research, learning, planning, reading, or avoidance depending on the task.

Human-AI interaction

Human-AI interaction research helped me question how confident Focus Integrity should appear to be. If Ticker gives a score without showing uncertainty, the interface may feel more objective than it actually is. Amershi et al. (2019) argue that AI systems should make capabilities and limits clear, support correction, and adapt cautiously. This connects directly to my correction-label idea and to the next step: clarification questions when the evidence is ambiguous.

Surveillance and workplace monitoring

The surveillance literature made the project ethically more serious. Ajunwa et al. (2017) show how worker monitoring can become expansive when productivity behaviour is turned into judgement. Urquhart et al. (2022) also show that AI-enabled workplace surveillance raises questions about privacy, agency, and trust. This matters for Ticker because Focus Integrity could easily become a system that watches more simply because more signals are technically available.

Why this is not just a machine learning problem

Focus Integrity cannot be treated as a simple classification problem because the labels are not stable. “Focused” or “not focused” depends on task context, intention, tool choice, working style, and sometimes the user's internal state. The same behaviour can mean different things in different contexts, so the design problem is not only how to train a better model. It is how to define, explain, and contest the judgement before the system acts on it.

Preparation

Week 8 pivot: from detector to task-aware AI senior

Week 8 will not be another scoring experiment. It will be a framing and communication week. I need to prepare Crit 2 by showing the pivot clearly: Ticker is moving from a focus detector to a task-aware AI senior. The next prototype should show how the AI understands the task before the session starts, creates a focus contract, and asks clarification questions when the evidence is ambiguous.

Week 8 preparation plan

Conclusion

Week 7 showed me that Focus Integrity cannot be designed as a simple accuracy problem. Users were willing to accept some monitoring when it helped them stay accountable, but they became less comfortable when the system could not explain why a score dropped. The most important finding was that trust depends on explanation, not only detection.

A lower-evidence model may be more acceptable if it is transparent, while a stronger evidence model may become too invasive if users cannot control what is captured. This shifts my next step from “make the model more powerful” to “make the judgement more explainable, contestable, and privacy-aware.”

Trust depends on explanation, not only detection.

References

• Ajunwa, I., Crawford, K., & Schultz, J. M. (2017). Limitless worker surveillance. California Law Review, 105(3), 735–776. https://doi.org/10.15779/Z38BR8MF94
• Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3290605.3300233
• Design Research Practice. (2026). DES303 Week 7 2026 [Course handout, University of Auckland].
• Dey, A. K. (2001). Understanding and using context. Personal and Ubiquitous Computing, 5(1), 4–7. https://doi.org/10.1007/s007790170019
• Dourish, P. (2004). What we talk about when we talk about context. Personal and Ubiquitous Computing, 8(1), 19–30. https://doi.org/10.1007/s00779-003-0253-8
• Nielsen, J. (2000, March 18). Why you only need to test with 5 users. Nielsen Norman Group. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
• Nielsen, J. (2012, June 3). How many test users in a usability study?Nielsen Norman Group. https://www.nngroup.com/articles/how-many-test-users/
• Urquhart, L., Laffer, A., & Miranda, D. (2022). Working with affective computing: Exploring UK public perceptions of AI enabled workplace surveillance. In Proceedings of Ethicomp 2022, University of Turku (pp. 164–177). https://doi.org/10.48550/arXiv.2205.08264
• Whitenton, K. (2021, February 21). Triangulation: Get better research results by using multiple UX methods. Nielsen Norman Group. https://www.nngroup.com/articles/triangulation-better-research-results-using-multiple-ux-methods/

Note on figures: Custom diagrams, reconstructed evidence views, and rating charts are based on the author's Week 7 development logs, replay tests, labelled segments, and peer testing notes. Figures 4, 7, and 10 are screenshots from the author's Ticker / Focus Integrity prototypes. Experiment 2 participant photographs are author-captured test documentation with face areas blurred before publication. All diagrams and prototype screenshots were built and rendered by the author, 2026.