WSJ R&D: bringing data science and AI capabilities to the newsroom

The Journal’s R&D unit works across the newsroom to develop new computational approaches to empower journalists with machine learning and artificial intelligence resources.

The Wall Street Journal Research & Development unit was launched in March 2018. Recognizing the impact artificial intelligence will play in journalism, the WSJ R&D (previously known as WSJ Media Science Lab) is a multidisciplinary team within the Wall Street Journal specifically focused on supporting the newsroom. R&D is part of the strategy group and collaborates with departments from across the Journal. Besides the full-time staff, the team hosts visiting fellows including academics, researchers and graduate students.

The output of the group includes newsgathering tools, data science driven stories and automation projects. Some recent efforts include a machine learning powered content analytics platform that analyses the Journal’s archive and helps journalists find editorial insights; an online tool that automatically turns college rankings data into text content and a story using computational journalism techniques to investigate Facebook’s data privacy sharing practices.

WSJ R&D, which is comprised of data scientists, machine-learning engineers, project managers and specialized editors, is focused on deploying new technology to help transform the newsgathering. It is centred on the following key activities:

• Develop AI tools to source information and story ideas through new types of data collection processes.
• Help journalists explore opportunities to dynamically redefine content packages at scale.
• Conduct research to identify technological trends that will impact the future of journalism.

“We build machines that help journalists find and produce powerful storytelling”, summarizes Francesco Marconi, WSJ R&D Chief and who is also the author of the upcoming book Newsmakers: Artificial Intelligence and the Future of Journalism.

In addition, the team develops newsroom guides for journalists including “how to cover the algorithmic beat” and “how to detect deepfakes”. Some of this research leads to broader initiatives such as the launch of the WSJ Media Forensics committee — a joint effort between the Standards & Ethics team and R&D to train journalists to detect AI-fuelled misinformation.

A shared process: increasing internal engagement

Marconi suggests that one of the strengths of the R&D team is its collaborative approach and the fact of being a shared resource. “We are trying to make the process as transparent and collective as possible within WSJ. We prioritize efforts to work on based on the strategic priorities defined by the newsroom leadership.”

This shared approach is connected with an internal platform for crowdsourcing ideas, which was implemented in the very beginning of the unit.

“We set up an idea submission platform to give everyone an opportunity to participate in the digital transformation process. The tool is nowadays a very important source of innovation for the broader newsroom, so much so that there will be a team dedicated to manage that process.“

Marconi explains that the ethos of the overall strategy department, which is led by Louise Story, is collaborative and inclusive. Anyone in the newsroom can submit an idea, and on a monthly basis there’s a review session open to anyone who wants to participate.

Besides the crowdsourcing platform, the R&D team sources new opportunities by hosting design-thinking boot camps for journalists, an activity run in close collaboration with WSJ’s Training & Outreach and UI/UX teams.

Prioritization principles: innovation connected with OKRs

Relying much on processes, this is a results-oriented unit. Every project is connected with broader strategic goals, which include diversifying and growing new audience, as well as differentiating WSJ’s journalism.

“We are not doing things just for the sake of experimentation. Everything we work on must be tied to a measurable outcome and, most importantly, have real newsroom impact. For every project we define OKRs (objectives and key results), a process that was introduced by our colleagues in the PDE (product, design and engineering) team”.

When defining the scope of new R&D undertakings, the team goes through a vetting and planning process to ensure projects satisfy the following guiding principles:

1. Newsroom impact: no unstructured experiments, every project must have a specific use case and outcome.
2. Collaboration degree: it should support editors and reporters across multiple departments.
3. Scalability: smart resource allocation, avoiding one-off projects.
4. Impact on newsgathering or production: the team’s goal is to help establish emerging journalistic processes, as opposed to new product development (an effort led by a different team).
5. Deployment of Artificial intelligence (AI) and/or machine learning (ML): those are considered the transformational capabilities necessary for the modern newsroom by the R&D unit

R&D in relation to the WSJ

R&D is designed to align with the overall strategy of the newsroom, and cooperation with other teams within the organization is therefore important and very much desired. One example is the data infrastructure provided by Dow Jones (WSJ’s parent company), which the group relies on in order to scale its projects.

“Something unique about developing AI capabilities at the Wall Street Journal is the support and guidance we receive from Dow Jones which has a fantastic technology team, as well as vast data science capabilities we can leverage.”

Besides teaming up with internal stakeholders, R&D also engages with universities and technology companies for select projects. For example, in collaboration with the Craig Newmark Graduate School of Journalism, the team tested an emerging newsgathering approach. They leveraged an AI-powered reporting device developed by Cortico, a “digital hearth” that dynamically records and structures interviews as the journalist guides subjects through questions about locally relevant topics.

As for future goals, Marconi explains that R&D’s ambition is to help democratize data science and AI practices throughout the newsroom while building on core journalistic skills and talents.

“These smart tools can augment the work of our talented journalists, but they can’t replicate their journalistic instinct. Although technology is increasingly impactful in the modern media landscape, the editorial process will always be driven by humans,” Marconi said.

CONTACT POINT

Francesco Marconi, WSJ R&D Chief

Twitter: @fpmarconi

Author