EXCLUSIVE: The film business has been through various revolutions in the last century: color, Cinemascope, THX Sound, 3D, premium seating and Large Format have taken the cinema experience to another level.
However, there’s one sector of the industry that hasn’t evolved and remains stuck in the stone ages: test screenings.
Yes, that post-production process by which studios assess the strengths and weaknesses of their films before a recruited audience only to have studio executives and filmmakers pull their hair out. To give you an idea of how long test screenings have been going on, consider the fact that Frank Capra was dealing with walk-out audiences and poor scores from the cards around 1936, when he showed off a three-and-a-half hour cut of Lost Horizon in Santa Barbara.
Do you know what the major revolutions have been in the feature film test screening business since Capra’s time? Screen test audiences no longer use Golf pencils, they use pens. The surveys went from being on one side to two sides of a piece of paper. While Netflix assesses its audiences via sophisticated algorithms, test screenings results for major studio movies are still being tabulated by hand, riddled with errors.
But now, the Edward Norton-backed analytics company EDO is catapulting the screening test process to the 21st century: They’re taking it digital, and studios including Warner Bros and other majors are on board.
No more papers, no more pens. If you are a participant at an EDO-run test screening, reach under your chair, and you’ll find a digital device the size of iPhone in a blue Ziploc bag. The new digital-questionnaire means of collecting audience data compiles statistical results within minutes versus the hours it takes to count paper ballots by hand. Not to mention, the results are more accurate and may even make the entire “focus group” concept extinct. Studio research folks often get nervous about asking too many questions to a participant during a paper test screening for fear that they wouldn’t answer everything. As such, they’d confine their questionnaire to two sides of a paper. With the EDO digital test screening, a studio can load as many as five to six pages of deeper questions and the participants will still yield a 100% completion rate.
“With the old two-sided paper card, the audience member would scan the card and take a ’buffet’ approach — they would skip around and leave a lot of questions blank, about 20%-40% of the cards just looked unfinished and empty,” he added. “Since we are only giving the audience member one question at a time, we get ’single question focus’, and we get a 100% completion rate on the closed end (multiple choice) questions. On the open-ended (short answer) questions, we have much more data in the responses. People are accustomed to using a digital device such as ours, and it’s not uncommon for us to get three to four times the length of response for the open-ended responses.”
Said Warner Bros Pictures Group chairman Toby Emmerich, who is fan of the product: “There’s always the question of how do you get the ‘very goods’ to go ‘excellent,’ or the ‘goods’ to ‘very good’ in the test screening process. With EDO’s device, you can ask a specific question to those who rated the movie as ‘good,’ or those who answered ‘very good,’ and quickly slice that data. It’s amazing how instantly you get the score and how complete the answers are. A lot of that has to do with the notion that people are use to typing on a keyboard versus writing by long hand.”
The EDO digital test screening device also uses “skip logic” in its questionnaire, which creates “nets” to catch data from specific parts of an audience. This assists the filmmaker in gaining greater insight in terms of what parts of the movie may or may not be working. Essentially, depending on the way you answer a question, it can lead to a subset of other questions.
“With this technology you’re only asking questions to the audience members you want to talk to, and you can treat the whole audience like a focus group — without the bias you get from the pack mentality of a focus group,” McLay said.
For example, earlier this year, EDO was testing a drama romance that had a Sixth Sense-type of reveal at the end. The first screening didn’t land well with the audience, which was thoroughly confused. The filmmakers addressed the problem and a second screening was held five weeks before the pic’s wide release. Essentially, the filmmakers added the following question to the questionnaire: “When did you think Character X was dead? A.) The Beginning B.) The Middle C.) When she looks back and sees he is not there.”
In regards to the skip logic of the questionnaire, if the audience member answered correctly (answer C), they then just moved on to the next question. However if they answered A or B, EDO gave them a follow-up question that asked, “What tipped you off that he might be dead?”
“If we had put that follow-up question on a paper questionnaire, the audience members who had answered the question right would have looked at the follow-up question and would think they missed something and would start changing their answers. The result would be a lot of junk data. With our system, the skip logic allows you to ask questions to only the audience members you want to talk to (in this case the audience members who marked A or B),” McLay said.
First, there’s nothing necessarily statistically viable about the focus group process. Cinema-screening pollsters created it to keep the filmmakers busy for the 45-minute duration when the test-screening results are being counted. Focus group opinions can also be erroneous: “People do as they do, not as they say they do,” McLay said about human behavior. Essentially, when asked about their opinion before a group, participants have an inherent nature to sometimes change their responses, as opposed to expressing themselves sincerely and privately on a polling device.
Nonetheless, EDO has kept the focus group intact at the client’s request because some filmmakers still enjoy it.
“With the billions of dollars invested in movies, it boggles the mind that when it comes to testing methodologies, the industry is still using paper and pen,” said three-time Oscar nominee Norton, who is not only an owner of EDO but also a client, having recently conducted a digital test screening for his upcoming Warner Bros crime drama directorial debut Motherless Brooklyn. “The software allows for so much more sophistication and speed … not to mention the nuance and depth of the way the people respond to the survey.”
Why did it take so long for the testing process to go digital?
“None of the old-system companies were willing to invest any money in innovation, so we thought there was an opportunity,” said McLay. “When I was at NRG I would look around the theater and watch people filing out cards using their thighs as a desk and it just seemed so inadequate; we’re using smart phones in every aspect of our lives, why not here?”
“This isn’t hacking a math problem, we’re trying to help creative people and studios who finance their work to do better and make interesting content,” Norton said. “Improved data helps everybody do better, and it flows down a chain with the studios ultimately feeling more comfortable taking risks and making more interesting movies.”