AI Agent for Literature Review: What It Can Automate, and What It Cannot

If you have done one of these before, you know the shape of it. The search returns more than anyone can read. You skim titles until titles stop meaning anything. Months later a reviewer asks about a source you did not cover, and you go back in.

I have not done that work, so I went looking for the measurement of it. In a sample of 195 medical systematic reviews registered on PROSPERO, analysed by Borah and colleagues in BMJ Open in 2017, the median review ran 65.8 weeks from registered project start date to publication, had a median of 5 people on it, and ended up including a median of 0.93% of the records its de-duplicated search returned.

Under one per cent. At that median, a review screened roughly a hundred records for every one it kept.

That is where the hours go: into ruling out papers you will never cite. Ruling things out against written criteria is mechanical and repetitive, which makes it the part software can take off you.

Below is the pipeline, stage by stage, with what I would hand over and what I would keep. I read the PROSPERO analysis and the pricing pages for Elicit, Covidence and Rayyan on 27 September 2026. I have never run a systematic review and I have benchmarked no screening tool, so there are no speed or accuracy numbers here.

A literature review is a pipeline, not a task

Almost every page on this topic treats the review as one request you make of a chatbot. It is eight jobs in sequence, and they have different owners.

1. Query definition. Turning your question into search strings, synonym sets and date limits.
2. Searching. Running those strings against every source you are entitled to search, and recording what each one returned and when.
3. Deduplication. Collapsing the same paper arriving from three places, and deciding what a preprint, a conference paper and the journal version count as.
4. Title and abstract screening. The big pile: include, exclude or unsure, with a reason attached.
5. Full text screening. The smaller pile, where criteria that need the methods section get applied properly.
6. Data extraction. The same fields out of every surviving paper, into one table.
7. Thematic synthesis. What the set of papers argues, including where it disagrees with itself.
8. Currency maintenance. Keeping all of that true through the months between finishing the search and defending the write-up.

Stages two, three, four and six are rule-following. They reward consistency and punish fatigue, which is a good description of what software is for. One and seven are judgement. Five is both.

Stage eight is missing from most descriptions of this work, because it is less a stage than a panic in the fortnight before submission. It is also the only part that recurs by nature, and a recurring job with a schedule on it is what separates an agent from a good chat session. For the tool category rather than the process, the AI research assistant landscape is its own piece.

Stage by stage, and where it breaks

This is the table I would want before setting anything up. The third column matters more than the second.

My own framing of the pipeline, written from the process rather than any tool's feature list. No throughput or accuracy figures appear, because I measured none.

One pattern in that third column is worth naming: the agent fails by producing something confident and wrong rather than by producing nothing, which is the failure mode that survives a skim. My read is that most of those failures trace back to a criterion written loosely.

So the hour you spend writing the criteria buys more than the tool you pick.

The extraction table

The extraction schema decides whether the output is usable, and it is the part almost everyone skips. Ask for the key findings of each paper and you get rows of prose in as many shapes as there are papers, which you cannot sort or compare. Ask for named fields with fixed values and you get a table.

A field needs four things specified: its name, its type, the permitted values or format, and what gets written when the paper is silent. Here is a worked schema for an empirical review. Adapt the middle rows to your field and leave the housekeeping rows alone.

Four rules keep that schema honest past row fifty.

Controlled lists beat free text. If a field can be a list, make it a list. A free-text design column gives you "randomised controlled trial", "RCT", "randomized trial" and "experimental" for the same thing, and now you cannot count.

Absence needs its own token. not_reported, unclear, none and not_applicable are four different facts about a paper. An empty cell is none of them, and in three months you will not remember whether the paper was silent or the run failed.

Provenance is a field, not a courtesy. The source_location column is what makes a row checkable by someone who is not you, and it is most of what an audit trail is.

Verbatim for anything you will quote. Paraphrase is where numbers drift.

Then run the schema on five papers you know well enough to grade it against, and compare cell by cell. The disagreements show you which field definitions are ambiguous. There are always more of them than you expect.

Keeping it current while you write

This is the stage nobody writes about, and the one an agent is actually shaped for.

Look again at that 65.8 week median from the 2017 analysis, measured from registered project start date to publication across 195 medical systematic reviews. Your search finishes early in that window. Everything after it is writing, revising and waiting, while the literature carries on without you. By the time anyone reads the review the search is old, and the fix is either a rushed re-run in the last fortnight or a reviewer noticing first.

A scheduled job replaces that with something boring and continuous. One saved description, running weekly, producing four things:

• New records matching the search, with the date and the source they came from.
• Which of them pass your inclusion criteria, and why, so you triage a short list instead of reopening the pile.
• Changes to papers already in your table: a correction, a retraction, a preprint now published with different numbers.
• An explicit "nothing new this week" when there is nothing, which is a result.

That last line does more work than it looks like. A dated series of re-runs, each with its query, its source and its count, is the search history your methods section has to state anyway. The maintenance and the paperwork turn out to be the same artefact.

Gravity runs jobs of that shape: you describe the recurring task in a sentence and an agent runs it on schedule. If you would rather wire the scheduling yourself, scheduling agent runs with cron covers the mechanics.

Two limits. What the weekly report sees is bounded by the sources you can point it at, and criteria drift as you read, so re-read your inclusion rules every few months.

What an agent must not decide

Four decisions here have to carry a person's name. Not because the software is bad at them, but because these are the ones you will be asked to defend, and "the agent judged it that way" is not an answer.

Whether a study is any good

An agent can tell you a paper reports a power calculation. Whether that calculation is credible, given what the field knows about effect sizes here, needs someone who has read enough of the field to hold an opinion. Appraisal is also the first thing an examiner probes.

Risk of bias judgements

Structured appraisal tools look like checklists, and checklists look automatable. Each item is really a question about whether the method the authors describe plausibly happened as described, which is an inference about a document and beyond what extraction can give you. Let the agent gather the sentences each item needs. You answer the item.

Which borderline papers get in

The unsure pile is where the scope of the review actually gets set. Each borderline call is a small decision about what the review is about, and together they define the boundary you defend in the methods section. Hand them over and you cannot explain your own inclusion logic, because you never formed it.

What the literature adds up to

Summarising a set of papers and saying what they collectively show are different acts. The value sits where studies disagree and someone works out why: different populations, different instruments, a design that cannot support the claim built on it. An agent smooths that flat, because a smooth summary reads better.

One rule covers all four. An agent may search, sort, extract and report. A person decides and signs. The AI agents for PhD students post takes the same line across the rest of a doctorate.

Systematic reviews are a different job

If your review is registered, the reporting requirements are the job, and that changes what tooling you should buy.

PRISMA 2020 is a reporting standard. Its own site describes it as a checklist of items and sub-items plus an expanded checklist detailing reporting recommendations for each item, distributed under CC BY 4.0, with a Shiny app for completing it, as of 27 September 2026. Read the checklist itself rather than anyone's summary of it, mine included.

No pipeline becomes compliant with a reporting standard by running. A standard describes what your write-up must state: what was searched, when, and what was excluded and why. An agent that cannot produce that log is unusable for a registered review however well its summaries read. An agent that can produce it has done most of the tedious part.

Purpose-built review tools exist largely to produce that trail. Covidence's Single plan was $339 USD a year for one review, with unlimited invited collaborators and valid for 12 months, when I read its pricing page on 27 September 2026. Rayyan offers a free signup on its own pricing page and advertises ResearchPilot and a risk-of-bias feature as new; its plan prices did not render for me, so I have none to quote and will not invent one. Elicit has a free Basic tier for casual exploration, and paid per-user tiers whose figure changes with the segment and billing period the page is set to, so read the toggles before you budget. Its higher tiers advertise a dedicated systematic review workflow that can screen 5,000 papers, and its Enterprise tier advertises 40,000 papers plus "PRISMA-grade screening and extraction accuracy". That phrase is Elicit's own marketing copy on its pricing page, and I have not tested it.

Registered review: buy the tool that produces the trail, and use an agent for the work sitting outside it, which is mostly stage eight.

Setting one up

Six steps, in order. Five of them are preparation, and people start at the fifth.

1. Write the inclusion criteria on one page. Every criterion phrased as a yes or no question a stranger could answer from an abstract. If a criterion needs the methods section, mark it as a full-text criterion and keep it out of the abstract pass.
2. Assemble the sources you can legally reach. Several indexes publish documented public routes: NCBI describes the E-utilities as the public API to the Entrez system, covering PubMed and PMC; arXiv publishes an API help page; Semantic Scholar states that most Academic Graph endpoints are available without authentication; OpenAlex documents a REST API alongside its own access and pricing page. All read on 27 September 2026, and all of them facts about those services rather than about any agent platform. Whether you can query a subscription database programmatically depends on your institution's licence and that database's terms. That is your side of the job, and you should read the terms before pointing anything automated at it.
3. Size the job around the rate limits. Semantic Scholar's API page says its unauthenticated endpoints are limited to 1,000 requests a second shared across everyone using them, while the introductory limit for an API key is 1 request a second, which is slower per person than the headline suggests. NCBI allows a few requests a second, raised if you register a free key, and asks that large jobs run overnight. Plan the slow overnight pass.
4. Fix the extraction schema. Fields, types, allowed values, absence tokens.
5. Build a calibration set of five to ten papers you know well. Run screening and extraction on those, compare cell by cell, fix the definitions that disagreed, then run it wide.
6. Schedule the re-run and name the signer. Weekly is enough for most write-ups, and someone has to own the unsure pile even if that someone is only you.

On Gravity: it is in private alpha, so you apply first. You describe the recurring job in plain words and an expert-built agent runs it on schedule. Your first agent is free and needs no card. After that, Pro is $5 a month during the alpha and Max is $20 a month, and you can buy more usage if you run out. Current plans are on /pricing. A run that fails on a platform error, with no usable output, is refunded under the refund policy.

Compare the shape of the spend as well as the number. Covidence's Single plan was $339 USD for one review over 12 months on 27 September 2026, a per-review purchase. A monthly subscription running recurring jobs is a different kind of spend, and for a registered review it sits alongside a review tool rather than replacing one. If you are earlier than all of this, the AI agents for students guide starts further back.

FAQ

Sources