The Ethics of Product and Software Experimentation
June 30, 2020
In my short fifteen months as a Research Triangle area resident, it has lived up to its reputation as both a tech startup hub and a center of medical research. In fact, this reputation is why my fiancee and I ended up here. Rachael is pursuing her Master’s and Ph.D. in clinical psychology at UNC-Chapel Hill, and I work in product at Pendo. When we get our friends together, it’s like we’re speaking different languages. I don’t understand the intricacies of what they do day-to-day, nor do they understand what it means to be a product manager. They don’t speak in KPIs, OKRs, or ARR.
However, we do speak one common language: that of experimentation. As product managers, we form hypotheses. We design and conduct experiments. We collect data, form conclusions, and communicate the results. By design, we follow the same scientific method used by clinical researchers.
To the psych students’ surprise, product managers use the same vocabulary as they do to diagnose patient issues and run psychological studies at a massive scale. It’s just focused on user behavior instead.
Research and regulatory bodies
When Rachael and her colleagues propose a new scientific study, it must be approved by the university’s Institutional Review Board (IRB). This lengthy process exists to protect the welfare, rights, and safety of the participants, and guarantee that the study follows federally mandated requirements. The National Research Act of 1974 established guidelines for studies on human subjects as a reaction to infamous experiments like Milgram’s shock therapy studies and Zimbardo’s Stanford prison experiment. In the pharmaceutical and medical device industries, such studies must be approved by the FDA in much lengthier and more rigorous processes.
Software and … product managers?
In some regards, product managers are researchers as well. We interview customers, consult users about their problems, and form hypotheses about the best way to improve their lives. Like Rachael and her psychology colleagues (or any principal investigator on a large scale research project), we use experiments to study human behavior.
Given these similarities, should we start framing our jobs like scientific studies and using the same terms to describe our work? Are there enough commonalities to justify the regulation of running “experiments” on human subjects? What if the purpose is to help them do their jobs faster and more efficiently? Do PMs need their own version of the Hippocratic oath taken by doctors?
That said, the nature of our “experiments” are different. In B2B software, they’re typically run on a smaller scale with lower stakes. The primary objective is to validate some hypothesis. It can be resource-intensive and slow to design the ideal experiment with the proper controls and variables, so we do the best we can with what we have.
To clarify, I’m not trying to argue that the software industry should be subject to the same processes and regulations as medical studies. I am, however, trying to make the case that product managers have a responsibility to be intentional, mindful designers of experiences and “experiments.”
The responsible product manager
With the absence of a regulatory body to manage the ethics of software feature experimentation, the responsibility to ethically validate hypotheses falls on the product manager and product designers. As the principal investigators of these research questions, we have an obligation and responsibility to protect the subjects of our experiments.
Perhaps when we design and implement these solutions, we can make what we do a little more ethically aware. For example, we might ask ourselves the following questions:
- Are customers aware that they are the subjects of an experiment seeking validation of an idea?
- Is this experiment being communicated clearly and openly?
- Did the customers consent to have their product experiences change for their benefit?
At the rate we need to work to grow a business and establish a new market, the only protection of our customers’ rights, welfare, and safety lies in our courage to ask the right questions, think through the implications, reach out to customers outside of our expected user groups, and constantly improve. It is our collective responsibility to consider the implications of the products we put out into the world.
Fri Oct 28
Thoughtworker Spotlight: Federico Galindo
His job title is Lead Product Designer, but his role stretches much more beyond those three words. Just ask any Connector, client, or practitioners from the community who’s met him - when Federico leads, people follow. Which makes sense when you consider the daily affirmation he sets for himself. World traveller, proud cat father, and avid musician (can you guess what instrument), everyone who crosses paths with this indelible teacher considers themselves the better for it. Discover this and more in this week’s Connector Spotlight.
Thu Oct 13
Mapping Your Product’s Ecosystem: A Product Strategy Essential￼
Product strategy should take all parties into consideration. It should account for the organization making the product, the environment it operates in, and the entities with whom they interact. It should assess and clearly communicate dynamic strategic threats while allowing for easy comparisons to determine differentiation. Last but not least, your product strategy should always put your customer first. Here's a how-to on plotting out the many actors, entities, and relationships involved in your forthcoming product to ensure your strategy leaves nothing to chance.