In a previous blog post, we discussed the use of ChatGPT for PubMed searches. Now that Gemini, previously Bard from Google, has become available in Canada, it’s worth taking a look at how the technology interprets requests for PubMed searches and how it differs from ChatGPT.

We’ll be using the exact same example that we used previously:

In former smokers with chronic obstructive pulmonary diseases, is pulmonary rehabilitation an effective treatment method?

When asked to generate a MeSH only search string, this is what the program comes up with:

("Chronic Obstructive Pulmonary Disease"[Mesh] AND "Pulmonary Rehabilitation"[Mesh]) AND ("Exp-Smokers"[Mesh] AND "Smoking Cessation"[Mesh]) [Optional: AND "Treatment Outcome"[Mesh]]

It seems like all AI programs share a brain cell when it comes to MeSH. Like ChatGP, Gemini is making up its own MeSH terms – Pulmonary Rehabilitation and Chronic Obstructive Pulmonary Disease are not MeSH terms! It has also included Exp-Smokers, when the correct MeSH term is Ex-Smokers. Yikes.

Another obvious issue is the use of AND to combine “Exp-Smokers”[Mesh] and “Smoking Cessation”[Mesh]. In the context of this question, these are synonyms, part of the same concept, and would therefore be combined using OR.

Let’s take a look at Gemini’s response as a whole:

While ChatGPT just spits out MeSH terms, Gemini seems to take it one step further and offer advice. The use of Required and Optional is presumptuous, but I can appreciate that the program tries to explain what kind of articles your query will return.

The biggest problem, and the one I take the most issue with, is the program’s declaration that adding the MeSH term Treatment Outcome will broaden your search. While Treatment Outcome might be a broad term, including this MeSH term will severely narrow your search and potentially eliminate relevant articles. The more concepts you combine using the Boolean operator AND, the less results you will end up with.

When asked to generate a list of keywords, Gemini provided one or two synonyms for each concept, even though we know from the complete search we generated in the last post that there are way more out there:

If you’re going to be using any AI program to generate synonyms, it’s best to enter your terms one at a time, and only after you’ve combed through other sources, like relevant journal articles or a thesaurus.

Gemini is also struggling with the concept of Booleans. The search string it generates is riddled with Boolean errors:

(COPD OR ex-smoker) AND pulmonary rehabilitation AND (exercise training OR dyspnea OR quality of life)

My head hurts just looking at this… I’ve used different colours to group together keywords of the same concept. COPD and ex-smoker are two entirely different concepts and should not be grouped together. If exercise training is often used in conjunction with pulmonary rehabilitation like Gemini says, then why is it not being combined using OR? Presumably, Gemini thinks that dyspnea and quality of life are important outcomes, so the use of OR to combine them is not false, but it shouldn’t have grouped exercise training with them. Overall, it’s a mess.

One thing that I think Gemini does better than ChatGPT is the explanation for searching both MeSH terms and keywords, and I trust it more as a learning tool than as a search string generator:

If you’re still not sure why you should complement your MeSH terms with appropriate keywords, take a look at our Health Sciences FAQ.

By now you’re probably thinking, “Gee, I don’t think Gemini could possibly get any more wrong.” Well, BUCKLE UP! I asked Gemini how to exclude keywords, wanting to know how it would explain the Boolean operator NOT:

Here, Gemini is telling you that instead of using NOT, you can simply use AND – to exclude a term. That’s not how that works! Booleans are sacred! They’re as old as time itself! You can’t just go around changing them! PubMed’s own User Guide warns that the minus sign (-) is converted to a space. Consequently, if you wanted to retrieve articles that talk mention COPD but not bronchitis, Gemini would have you type this: (“COPD” AND – “Bronchitis”). But PubMed isn’t interpreting this search the way Gemini thinks it is:

PubMed is ignoring your minus, turning it into a space, and searching instead for articles that mention both COPD and bronchitis. The correct way to exclude bronchitis is by entering: “COPD” NOT “bronchitis”.

I’m not ready to give you my blessing to generate searches using ChatGPT or Gemini. Maybe one day we’ll get there, but for now, I think I’ve demonstrated why you shouldn’t trust either of these tools with the job. The good news is that if you’re struggling with searching in PubMed or other medical databases, your librarians are here to help you! Contact us with any questions relating to searches and databases.