Two months in, what has the FT learned about launching its own generative AI tool, AskFT

An interview with Lindsey Jayne, FT Chief Product Officer

While many news media worry about disintermediation in the age of AI, there are also exciting developments occurring on publishers' own platforms. The conversation in fact often revolves around internal efficiency gains such as faster newsgathering or more sophisticated propensity models (which are important), but AI, and specifically generative AI, is enabling new ways to engage readers through dynamic experiences. Since the beginning of the year we’ve in fact heard various announcements from news media experimenting with chatbots, such as Ask Skift, Forbes beta-testing Adelaide, BloombergGPT, and more.

Two months ago, the FT also ‘entered the chat’, launching its own generative AI tool called AskFT.

AskFT is available for a subset of FT Professional (B2B clients) users designated as beta-testers, but how does it work? Once a user asks a question, AskFT finds an article segment from the FT’s archive dating back two decades that is semantically similar to the question. Following that an LLM model is used to summarise those exact articles into a generated answer. For now, it's a ‘one question, one answer’ tool, and the answers include citations that correspond to the specific FT article summarised by the model.

In this blog post we’ll talk to Lindsey Jayne, the FT’s Chief Product Officer about how the FT came up with AskFT, how they’re measuring success two months into beta testing, and what the future could look like.

Lindsey, thank you for joining us. Can you tell us about why the FT decided to experiment with launching a generative AI product, and what user needs you thought it could solve?

Generative AI is an example of the ‘chicken or egg’ dilemma between what a user needs and what technology can do. You should always start with a user need and build something to meet it, but sometimes technology comes around that can do things that weren't possible before, and you have to start testing the limits of that and understand it.

We were already working on understanding how we could make sure our Professional customers were getting more value out of the FT, such as helping them better understand emerging contexts of news events, being early on understanding how larger themes might land, or helping them to quickly get to grips with a new company or client they are working with.

Ultimately, users trust the FT’s take on something but the way they were getting that was by reading through everything and building the knowledge up, and there weren't really any tools to help them answer a question they had in mind. That’s the context behind AskFT.

We had built a semantic search, where we broke our articles down into segments and vectorized them, as a way of giving more tailored recommendations to users. In fact, as a side note, today FT Professional users can highlight a bit of text on any article and we'll give them a recommended read that's similar. So we were very fortunate in that we were already thinking about the opportunity, and we already had some of the underlying tech.

AskFT has been available to 500 Professional users over the past two months. These 500 users were picked at random from a group who were active enough for us to ask them to test it, to ensure we had a broad range of industries and roles that would represent our target audience. We are actively evolving it into a product that will be free for all FT Professional subscribers to use.

Why did you decide to go for it last year?

All LLMs still have a level of hallucination. As a news organisation that prides itself on trust and gold standard journalism, there was this interesting question of: do we even go there? We have clear principles on how we use generative AI, set by our editor, as well as standards on ethics and quality that we hold ourselves to in any product development.

But we had two reasons for why ‘yes let’s experiment’, namely:

1. We have to use it to understand it, so we need to test and practically learn the limits and opportunities
2. We shouldn't just build for what fast-moving tech can do today, we should build bearing in mind where the tech is going

So we spent some time experimenting to see what we could do. And Ask FT was one of those experiments. We had it up and running quite a while before we released it to customers, because we wanted to be really, really sure it was good. Obviously you will not remove all risk of hallucination, of model drift, and we make very clear to customers that this is a risk, and that the answers are machine generated and not fact checked. But we also found lots of checks and balances we could put in place so the output was useful, for instance, the risk was lower with Retrieval Augmented Generation (RAG) than it is with purely generated information.

We’re also very fortunate to have a dream team to assess how good the tools is. Not just the talented folk in machine learning, product and technology, but a whole newsroom full of exceptional people who fact check and edit for a living. They gave us their time to help make sure this was worthy of going out onto the FT.

Talking about this experimentation phase, what was the model trained on?

Let’s talk about the model and then the data we use, because those are two separate things.

First, the model. We experimented with a number of them. I was keen that we were model agnostic, that we should test and use whatever was the best tool for the job, and that we should be prepared to switch models, or use different models for different parts of the task. And so the team has developed a framework to assess quality for our specific needs.

Second, training. We have not trained a model on our content. Instead, we select segments of articles that are semantically similar, or similar in meaning, to the question asked. We then ask the model to summarise an answer to the question based on these segments, and cite them. We have a number of other steps and tests. We have models run to ensure the quality of the answer, but that’s our broad approach to get a quality answer without going through training a model.

You mentioned hallucinations before and wanting to make sure an FT product was up to standard for the brand. How did you go about testing that?

Initially we tested the tool internally, making it available to the whole company. We also built a multidisciplinary group including product, technology, commercial, and editorial and gave them specific instructions on the hallucination risks we were looking for. We asked them to run tests and then we recorded all of the answers and iterated our prompts and code based on it. We also had a third party company try to jailbreak the tool.

AskFT has been live for about 2 months, how are you measuring success of the tool and what have you learned so far?

For us it all comes back to the value or utility to our users, which means we start by asking if it’s helpful. And even in this minimum viable form, 75% of people have said they found it helpful. There a number of metrics that complement this overall idea of usefulness that we also track:

1. Whether they go one to read the articles cited
2. When and how they come back to the tool
3. How performant or fast it is (there is still some loading time with these tools)

We also follow what sorts of questions people ask, and the answers they get back, both so we can audit for quality, but also so we can understand what sorts of things people are finding valuable. We have a survey for repeat users to help us understand what they were hoping to get out of the answer, and we can map that against their industry and role to see how best to help them do what they need to.

In this first minimum viable product, we didn't embed AskFT into our website or app, so it was only available through a specific link that you had on an email. This means that beyond some power users, people haven’t yet built a habit around using it. Our next iteration will start to embed the tool more naturally into people’s reading and workflows, so it’s there when you need it.

What are the plans for AskFT’s future applications and its overall evolution?

Internally, we’ve actually ended up using some of the capabilities that underpin AskFT in tools we’re building for our newsroom. For example, AskFT is a much more intuitive and powerful way to search our own archives and pull together history than the tools we have available.

What I can say further today, is that as we incorporate user feedback and move to a fully fledged product, we’re starting with looking at things like saving searches, being able to refine an answer, as well as speed and quality improvements. For our Professional audience, we have some great things in development that I can’t share just yet, but do watch this space as it’s really exciting!

If you would like to learn more about our AI solutions and how FT Strategies can help implement AI across your organisation, please get in touch here.