BioMed LLM

Problem and users

A language model only knows what was in its training data. This project tries a simple retrieval step for biomedical question answering: look up a related PubMed article when the question is asked, and add it to the model’s context.

It is a research notebook, not a tool for medical decisions.

How it works

1. Question and contextThe user supplies a yes, no, or maybe question and a passage of context.
2. PubMed searchEntrez esearch runs the question text against PubMed and keeps the single top result.
3. Retrieved textEntrez esummary fetches that article's title, which is appended to the supplied context.
4. GenerationBioGPT completes a fixed prompt with greedy decoding, up to 50 new tokens.
5. AnswerThe Gradio interface returns yes, no, or maybe by looking for those words in the generated text.

The model is the microsoft/BioGPT-Large-PubMedQA checkpoint from Hugging Face. The notebook builds this prompt and truncates it at 512 tokens:

question: {question} context: {context} {retrieved title} the answer to the question given the context is

The retrieved title is the only text the notebook adds from PubMed. The interface does not show it as a citation, so an answer cannot be traced back to a source from the interface. The generated answer is not evidence that the retrieved article supports it.

Evaluation and limitations

The notebook runs the pipeline over the 1,000 expert labeled questions in PubMedQA (pqa_labeled), in five batches of 200. An answer counts as correct when the expected label appears anywhere in the generated text.

I am not presenting the resulting accuracy as a finding, for four reasons:

• No baseline. There is no run without retrieval, so the notebook cannot show whether retrieval helped.
• Likely overlap with training data. The checkpoint was fine tuned on PubMedQA, and the evaluation uses the same labeled questions.
• Loose scoring. Substring matching can count “no” inside words such as “not” or “know”, and an output containing several labels can match whichever is expected.
• Thin retrieval. Each question adds one article title, while each PubMedQA example already includes its source abstract as context.

A stronger evaluation would use held out questions, compare runs with and without retrieval, and parse a single label from each output. Nothing here is medical validation.