SLReviewFlow
GuideRemoving duplicates

Guide · Searching and pooling

How to remove duplicates between Scopus, Web of Science and PubMed

The same paper turns up in every database you search. This guide shows how to export, match and remove the repeats, and how to report the count in your flow diagram, with or without SLReviewFlow.

By , Research MindUpdated

Start a reviewJump to Excel steps
Worked exampleillustrative numbers
ScopusCSV export412
WoSExcel, one file368
PubMedPubMed format255
Records identified1,035
Same DOI, PMID or accession number−290Removed without asking: the identifiers agree
Look alike, no shared identifier31 pairsYou decide: 24 confirmed, 7 not duplicates
Confirmed by you−24
Records to screen721Flow diagram: duplicate records removed (n = 314)

01Why the same paper appears several times

Scopus, Web of Science and PubMed index many of the same journals, so a search run in all three returns the same article more than once. Each copy looks slightly different: one has a DOI and another does not, author names are written Smith, J. in one export and John Smith in another, and titles differ in punctuation, capitals or a trailing full stop.

Whatever kind of literature review you are running, readers will want to know how many duplicates you removed before screening, so the aim is not only a clean list but a number you can report and defend.

02Export every record with its identifiers

Export each database in a format that keeps the abstract and the database's own record number. Those numbers make matching exact later.

DatabaseWhereChooseKeeps
ScopusExport, then CSVCitation information, Bibliographical information, Abstract and keywordsDOI, EID, PMID
Web of ScienceExport, then Excel or BibTeXRecord content: Full record. Large result sets export in several files; keep them all.DOI, UT, PMID
PubMedSave, then Format: PubMedSelection: All resultsPMID, PMC, DOI

Write down the date and the number of results each database reported. You need both for your flow diagram, and they let you check that nothing was lost in export.

03Match on identifiers first

Two records with the same DOI are almost always the same work, so start there. Clean the DOI before comparing: lower-case it and remove https://doi.org/ or doi: from the front. Then do the same with the identifiers each database adds.

DOI
10.1000/xyz.0142
Most journal articles; often missing on older records and conference papers
PubMed ID
PMID 38461094
In PubMed, and in Scopus and Web of Science for many medical papers
Scopus EID
2-s2.0-85187…
Only in Scopus; finds repeats across two Scopus searches
WoS accession
WOS:001184…
Only in Web of Science; the same for repeats across WoS files

Example identifiers.

Two records that share a DOI but have different years or titles are worth a look: it can be an erratum, a corrigendum or a retraction notice that carries the article's DOI.

04Then compare title, authors and year

Records without a shared identifier still need checking. Compare a normalised title (lower case, no punctuation), the first author's family name and the year. A close match is a candidate, not a duplicate: a conference paper and its later journal version, or a preprint and its published article, can share a title and still be reported separately.

Web of SciencePubMedOne paper, two exports, no shared identifiertitle, authors, year 0.93
Web of SciencePubMed
Title
Point-of-sale tobacco display bans and smoking initiation among adolescents: a systematic review
Point-of-sale tobacco display bans and smoking initiation among adolescents.
Authors
Mensah, D; Chou, M
Daniel Mensah, Mei-Ling Chou
Year
2026
2026
DOI
not in export
10.1016/j.ypmed.2026.10811…
CConfirm duplicateNNot a duplicatePKeep this record

Example records. Marked text is what differs. The keys are the ones SLReviewFlow uses in duplicate review.

05Choose which record to keep

When you confirm a duplicate, keep the most complete copy: the one with an abstract, then the one with a DOI. That kept record is what you screen, so a missing abstract now means a harder decision later.

  • Keep every removed copy in a log with the reason, so a reviewer can trace each number.
  • Do not merge fields from two records by hand; note which one you kept.
  • Ask a second reviewer to check a sample of close matches if your protocol says two people screen.

06Doing it in Excel

Excel works for a few hundred records. Paste every export into one sheet with the columns Source, Title, Authors, Year and DOI, then add a cleaned DOI column:

F2, then fill down
=LOWER(TRIM(SUBSTITUTE(SUBSTITUTE(E2,"https://doi.org/",""),"doi:","")))

Keep a copy of the original sheet. Sort by the cleaned DOI and mark repeats with Conditional Formatting, Highlight Cells Rules, Duplicate Values. For each group, choose the most complete copy to keep and move the others to a Removed sheet with the reason, so your log matches the count. Avoid Data, Remove Duplicates: it deletes every later copy at once and keeps whichever came first. Then sort the rest by title and read down the list for near matches.

Filter out blank DOIs first. Records without a DOI share an empty cleaned DOI, so a sort or a duplicate check groups them together as if they were one work. An erratum or retraction notice can also share its article's DOI; keep it.

Reference managers such as EndNote and Zotero also find duplicates; Bramer and colleagues describe a step-by-step EndNote method (2016).

07Report the count in your flow diagram

Most reporting guidelines ask for this number. In a PRISMA 2020 flow diagram it goes in the first column, under records removed before screening. In your methods, say how duplicates were found and which tool you used.

Records identified from
Scopus (n = 412)
Web of Science (n = 368)
PubMed (n = 255)
Records removed before screening
Duplicate records removed (n = 314)

From the worked example: 290 identifier matches plus 24 confirmed by you.

08Doing it in SLReviewFlow

Import the exports as they are, and only decide the close calls

  1. Add each search with its date and query, then upload its export: Scopus CSV, Web of Science Excel or BibTeX, PubMed, IEEE Xplore, OpenAlex or RIS.
  2. Build the pool. Records that share a DOI, PMID or database record number are removed automatically, and repeated imports of one search collapse.
  3. Review the records that only look alike with C, N and P, then screen. The flow diagram fills in from your decisions.
Start a reviewSee how it worksFree while in early access

09Questions

Should I remove duplicates before or after screening?

Before. Reporting guidelines such as PRISMA 2020 count duplicates removed before title and abstract screening, and screening the same paper twice wastes time and can lead to two different decisions on one paper.

Is matching on DOI enough?

No. Many records, especially older ones and conference papers, have no DOI in at least one export. Match identifiers first, then compare title, authors and year for the rest.

How many duplicates should I expect?

It depends on how much your databases overlap in your field. Use the numbers you found; there is no correct share.

Can I remove duplicates inside Scopus before exporting?

Not across databases. Within one database, export everything and let the record numbers find the repeats, so your count matches what the database reported.

10References

  1. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https://doi.org/10.1136/bmj.n71
  2. Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc. 2016;104(3):240–243. https://doi.org/10.3163/1536-5050.104.3.014 (erratum 2017)
Review guidesEvery guide on searching, screening and reportingAuthorKushan Liyana Arachchige's public profileHomeThe six stages of a review in SLReviewFlow