How the Strands Connect

Three independent data streams, each answering a different kind of question about the adoption journey, funnel into a single synthesis. The diagram draws itself as you scroll to it.

What Each Method Was Built to Answer

I organized my inquiry across four dimensions of the educator adoption experience: Confidence, Attitudes, Knowledge, and Application. Each method was chosen because it addressed something the others could not. Click or tap a segment to explore what each method covers.

Twelve Months of Data Collection

Click a phase to see what data collection and product research work I did at each stage.

Running the Numbers & the Themes in Parallel

Both strands ran simultaneously, each informing how I interpreted the other. The quantitative data showed what changed in educators’ work using the platform. The qualitative data told me why it mattered to the people living through it.

Quantitative

Instrument adaptation. I worked with the team to map T-STEM’s existing items to the Fellowship’s content domains (safety science, sustainability, Action-Oriented Pedagogies, and platform resources) and co-wrote new items for constructs the original scale didn’t cover. This required reading the program’s logic model alongside the instrument’s validation literature to make principled decisions about what to preserve and what to adapt.

Statistical pipeline. After discussing with my supervisors and presenting the analysis techniques I would suggest, I went on to use Welch’s independent samples t-test because it does not assume equal variance, which of course was a more defensible choice for this participant profile than the standard alternative. Before running any inferential tests, I generated descriptive statistics and exploratory boxplots to examine distributions and flag anomalies that could affect interpretation.

R environment: survey data pipeline, Welch’s t-test code, and a pre/post confidence boxplot. Statistical values redacted.

Qualitative

Reflexive thematic analysis. I applied Braun & Clarke’s six-phase framework to all focus group transcripts. This requires deep engagement with the data at every step and a willingness to return to earlier phases when new readings reveal patterns underweighted the first time.

1. Familiarization: Multiple transcript readings before any coding. Resisting premature closure.
2. Initial coding: Line-by-line inductive coding across the full dataset.
3. Generating themes: Organizing codes into candidate themes that captured dataset-wide patterns.
4. Reviewing themes: Testing each theme against the data; merging where distinctions weren’t analytically meaningful.
5. Defining & naming: Articulating the core meaning of each theme precisely enough to write about consistently.
6. Writing up: Producing the analytic narrative organized by evaluation question.

Artifact analysis served as the third lens, coding educator-produced instructional documents across multiple versions per participant to trace how their platform integration evolved. When survey and interview data told different stories, the artifacts provided the third point of evidence that resolved the tension, or confirmed that both patterns were real in different participant subgroups.

The Translation Problem

The hardest part of product evaluation is not analysis. It is translation, the work of converting patterns observed in data into recommendations specific enough to be actionable yet grounded enough to be credible. The distance between “here’s what the data shows” and “here’s what the program and product team should do differently” is where most evaluation reports quietly fail.

I approached this by organizing the synthesis around each evaluation question, assembling evidence from all three strands before drawing any conclusion, then framing recommendations explicitly in the program’s own logic model so stakeholders could engage with them without having to re-learn the research vocabulary.

Recommendations were written to influence program strategy, naming what to change in the platform’s onboarding model, what to sustain, and what to examine more closely in the next adoption cycle. The goal was to give decision-makers evidence they could act on, not just a report they could file.

Presenting to the People Who Make the Decisions

The final evaluation was prepared for product and program leadership. This required a deliberate translation step, moving from the vocabulary of research methods to the vocabulary of product design and program improvement. Importantly, different participants applied what they learned from the fellowship in their unique context, as such, the presentation images (left panel) while still about safety science, does not immediately look like it without checking other slides.

Giving a presentation as AZSTA, Mesa, Arizona.

Also presenting alongside participants at NSTA in Philadelphia.

I focused the stakeholder presentation around the evaluation questions and logic model that had framed the work from the beginning. Where the data told a more complex story than a clean summary would convey, I made that complexity explicit and showed what it meant for future product and program decisions in the form of recommendations. Unfortunately, I cannot share the findings and will leave this section here.

From the Field

Hover any image for context. Participants’ faces obscured where applicable.

What I Carry Forward

Product evaluation is user research with higher stakes and a harder audience. The participants are the users. The platform is the product. The logic model is the research question. The report is the deliverable. The job, in every case, is to understand people well enough to say something useful about them, and honest enough to say it even when it complicates the story the product team came in hoping to hear.