M4CXR Multitask Chest X-Ray Interpretation with Multimodal LLMs — IEEE Transactions 2025 Base Paper Project

A multimodal large language model (MLLM) is a language model that also reads images, so it can reason about what it sees and answer in words.

M4CXR is a multimodal large language model that interprets chest X-rays across three tasks: it drafts radiology reports by listing findings, grounds report phrases to image regions, and answers questions about the image. This project implements that system for a student build, evaluated on the public MIMIC-CXR test set, with an Indian hospital reporting assistant as the student's own addition.

Chest X-ray interpretation is a multi-task job in practice: a radiologist writes a report, points to where a finding sits, and answers questions about the image, often across several images or a prior study. Earlier language-model approaches to chest X-rays tended to do only one of these jobs, or to use a language model only for report writing and leave its other abilities unused. The paper by Park, Kim, Yoon, Hyun and Choi, published in IEEE Transactions on Neural Networks and Learning Systems in 2025, builds one model, M4CXR, that handles four "Multi" aspects together: multi-modal input, multi-task output, multi-image input, and multi-turn chatting. It is trained on a visual instruction-following dataset that integrates various task-specific datasets in a conversational format, and it uses a chain-of-thought prompting strategy in which it identifies findings in the chest X-ray image and subsequently generates the corresponding report. Its authors report state-of-the-art clinical accuracy in medical report generation, visual grounding at a level comparable to specialised models, and outstanding performance in visual question answering — those are the paper's own claims, labelled as such, and no numeric figure is quoted here because none is verified for this listing. This project implements that system for a student build, evaluated on the public MIMIC-CXR test set for report generation, visual grounding and visual question answering, and adds a clearly separate Indian contribution: a hospital chest X-ray reporting assistant with abnormality-grounding overlays and a plain-language English and Hindi patient summary alongside the formal report.

This is a built-to-order software project: the multitask chest X-ray interpretation system, demo console, report, presentation, and viva Q&A are prepared fresh for you. Starting price: Rs 28,000 (final price confirmed after topic, base paper and compute scope are locked).

This project is an academic prototype and is not intended to replace certified medical equipment, professional diagnosis or clinical decision-making. The demo console on this page uses an illustrated schematic, not a patient radiograph, and its draft text is demo text only.

The Problem

A chest X-ray service faces three linked jobs, and splitting them across three separate tools loses the connection between them. A report that says "opacity in the lower zone" is far more useful when the reader can see exactly which region that phrase refers to, and a clinician's follow-up question — "is there fluid here?" — should be answered from the same image and the same findings, not from a different model that never saw the report. Single-task models break that chain: a report-only model cannot ground its own phrases, and a general question-answering model has no radiology report to anchor its answers. Training one model for all three jobs is harder, because the training data must cover report writing, grounding and question answering in one conversational format, and because a report written in one pass can drift away from what the image actually shows. The paper's answer is to train a single multimodal model on integrated visual instruction-following data and to write reports by chain-of-thought — findings first, report second — so the report is built from findings the model has already committed to. The student build reproduces that reasoning chain on the public MIMIC-CXR test set, then faces the variant the paper does not cover: an Indian hospital reporting workflow where the same system must show a grounding overlay for a finding and give the patient a plain-language summary in English or Hindi alongside the formal report, without ever presenting demo output as a diagnosis.

Base paper at a glance

Base paper

• Paper: "M4CXR: Exploring Multitask Potentials of Multimodal Large Language Models for Chest X-Ray Interpretation", Jonggwon Park, Soobum Kim, Byungmu Yoon, Jihun Hyun, Kyoyun Choi, IEEE Transactions on Neural Networks and Learning Systems, 2025, vol. 36, no. 10, pp. 17841–17855.
• DOI: https://doi.org/10.1109/TNNLS.2025.3587687 (IEEE Xplore document 11106750; free preprint, separate from the DOI record: https://arxiv.org/abs/2408.16213).
• Problem the paper tackles: previous language-model approaches to chest X-ray analysis either underused the multi-tasking capabilities of large language models or lacked clinical accuracy, so one model rarely covered report generation, grounding and question answering together.
• What is implemented from the paper: a vision encoder plus projector feeding a large language model; training on visual instruction-following data integrating task-specific datasets in conversational format; chain-of-thought prompting that identifies findings before generating a report; medical report generation; visual grounding; visual question answering; and multi-image, multi-study and multi-turn handling.
• What is deliberately not claimed: no numeric accuracy, F1 or benchmark figure is quoted, because none is verified for this listing; the paper's "state-of-the-art clinical accuracy" claim is quoted only qualitatively as the authors' own claim; and no real patient image is shown in the demo — the console uses an illustrated schematic.
• Paper-reported results (the paper's own findings, not this build's measured results): state-of-the-art clinical accuracy in medical report generation through chain-of-thought prompting, visual grounding at a level comparable to specialised models, and outstanding performance in visual question answering. No numeric figure is attached to those findings here, because none is verified for this listing, and the student build's report records only the figures its own evaluation runs actually produce.

Own features

These are the student's own contribution, clearly distinct from the base paper:

• [Indian hospital chest X-ray reporting assistant] A reporting workflow layer over the M4CXR system: for a case, the assistant shows the formal draft report, a grounding overlay for a selected finding, and a plain-language summary. The paper builds the multitask model; it does not add this hospital reporting workflow. This is presented as a demonstrated scenario layer in the console on illustrated cases, not as a validated deployment on real hospital data, and it gives no diagnosis or treatment advice.
• [Plain-language English and Hindi patient summary] A short summary of the draft report in plain English, with a Hindi version alongside it, so a patient can follow what the report says in everyday language. The paper's scope does not cover bilingual patient summaries, and the summary explains demo report text only.
• [Interactive multitask demo console with visible chain-of-thought] A demo that shows the three tasks side by side — findings first, then the draft report; a finding clicked to its image region; a question answered from the same case — so a reviewer can see the multitask behaviour instead of taking it on trust. The console image is an illustrated schematic, labelled as such on the page.

The pipeline runs as a visible chain, and each stage's output is shown in the demo so a student can explain it in a viva. First, a chest X-ray image (in the demo, an illustrated schematic; in evaluation, a MIMIC-CXR test image) enters the vision encoder, and the projector converts its features into visual tokens placed among the text tokens. The language model reads the image tokens together with the task prompt — report, grounding or question — and, for report generation, first lists findings by chain-of-thought and then composes the report from that list. For grounding, a selected report phrase is mapped to an image region and drawn as an overlay. For question answering, the model answers from the same image and conversation history, over multiple turns if needed. The Indian hospital layer then presents the draft report, the grounding overlay and the plain-language English and Hindi summary together.

1. Take a chest X-ray image as input (illustrated schematic in the demo console; MIMIC-CXR test image in evaluation, subject to PhysioNet credentialing and a data use agreement).
2. Pass the image through the vision encoder to extract visual features.
3. Project the visual features into visual tokens and insert them among the language model's text tokens.
4. Read the task prompt together with the image tokens — report generation, grounding, or a question.
5. For report generation, identify findings first by chain-of-thought, then generate the report from the findings list.
6. For grounding, map a selected report phrase to an image region and draw the overlay.
7. For question answering, answer from the same image and conversation history, over multiple turns.
8. Handle single-image, multi-image and multi-study inputs in the same conversation.
9. Show the Indian hospital layer: draft report, grounding overlay, and plain-language English and Hindi summary together.
10. Evaluate on the MIMIC-CXR test set for report generation, grounding and question answering, and record only the figures the student's own runs produce.

Tech stack:

• Python 3 · PyTorch · Hugging Face Transformers
• Vision encoder · projector · large language model
• LLaVA-style multimodal framework · visual instruction tuning
• Chain-of-thought prompting · multi-turn chatting
• HTML · CSS · JavaScript demo console
• Matplotlib · evaluation plots
• Git · Jupyter Notebook

Dataset & model details

• Dataset (paper and student build): MIMIC-CXR, a large, de-identified, publicly available database of chest radiographs with free-text radiology reports from Beth Israel Deaconess Medical Center, distributed via PhysioNet. The student build is evaluated on the MIMIC-CXR test set for medical report generation, visual grounding and visual question answering. Access requires PhysioNet credentialing and a signed data use agreement, and the build's figures are its own and are not presented as the paper's results.
• Dataset (demo console): An illustrated chest schematic drawn for the demo console. It is not a MIMIC-CXR image and not a patient radiograph, and it is labelled as such wherever it appears. No real patient image is shown on this page.
• Dataset (hospital adaptation): No Indian hospital dataset is claimed. The Indian reporting assistant is demonstrated in the console on illustrated cases, not on a published Indian dataset, and it is labelled as such wherever it appears.
• Task: Multitask chest X-ray interpretation; input = one or more chest X-ray images plus a task prompt, output = a draft radiology report, a grounded image region, or an answer to a question about the image.
• Model: The paper's system — vision encoder plus projector into a large language model in a LLaVA-style framework, trained on visual instruction-following data integrating task-specific datasets in conversational format, with chain-of-thought finding identification before report generation. No architecture detail beyond the paper's description is claimed for it here.
• Metrics: Report-generation quality, grounding quality and question-answering quality as in the paper's multitask evaluation; the student build's report records only the figures its own evaluation runs actually produce, and no figure is claimed before those runs.

Multitask chest X-ray interpretation delivers seven working capabilities: a vision encoder plus projector into a language model, visual instruction-following training, chain-of-thought report generation, visual grounding, visual question answering, multi-image and multi-study handling with multi-turn chatting, and an Indian hospital reporting assistant, all visible in a demo a reviewer can operate. Findings are identified before any report sentence is written, so the report is anchored to findings rather than written in one unchecked pass, and the same model that writes the report can point to a finding and answer a question about it.

• [Vision encoder plus projector into an LLM] Chest X-ray images pass through a vision encoder and a projector that inserts visual tokens among the text tokens of a large language model, following the LLaVA-style framework used in the paper, so image and text are reasoned over together.
• [Visual instruction-following training data] Training data assembled by integrating various task-specific chest X-ray datasets into a conversational format, so one model learns report generation, grounding and question answering instead of one narrow task.
• [Chain-of-thought report generation] The model identifies findings in the image first and generates the report from those findings, the prompting strategy behind the paper's reported clinical accuracy in report generation.
• [Visual grounding] A phrase from the report is linked to a region of the image, so a reviewer can see where a finding is claimed to sit instead of reading the phrase alone.
• [Visual question answering] Questions about a chest X-ray image are answered from the same model and the same case, in a multi-turn conversation rather than a single one-off answer.
• [Multi-image, multi-study and multi-turn handling] The system adapts to single-image, multi-image and multi-study inputs, so a prior study or a second view can be part of the same conversation.
• [Indian hospital reporting assistant] A reporting workflow layer with grounding overlays and a plain-language English and Hindi patient summary alongside the formal draft report, demonstrated in the console on illustrated cases and kept visibly separate from the paper.

• Complete multitask chest X-ray interpretation pipeline source code (vision encoder plus projector, visual instruction-following training, chain-of-thought report generation, visual grounding, visual question answering, multi-image and multi-study handling)
• Interactive demo console (the web demo shown on this page) with report generation, visual grounding, and visual question answering plus the Indian hospital reporting layer
• Project report PDF (background, base paper, methodology, experiments, limitations)
• PPT presentation for final review
• Viva Q&A preparation document (multimodal LLMs, chain-of-thought prompting, grounding, MIMIC-CXR evaluation, hospital adaptation, limitations)
• Starting price: Rs 28,000 (final price confirmed after topic, base paper and compute scope are locked)
• Specific next step: message on WhatsApp for a quotation stating scope (M4CXR multitask chest X-ray interpretation with report generation, visual grounding and visual question answering on MIMIC-CXR plus Indian hospital reporting assistant with grounding overlays and plain-language English and Hindi patient summary), starting price Rs 28,000, and delivery under the 110-hour work plan as a built-to-order project

Work plan & timeline

Total work plan: 110 hours, split across six phases. Phase 1, base-paper study and environment setup: 15 hours — reading the Park et al. paper, setting up Python, PyTorch, and the image-text data pipeline, and completing PhysioNet credentialing for MIMIC-CXR access. Phase 2, vision encoder, projector and instruction data: 25 hours — the LLaVA-style multimodal framework and visual instruction-following data assembled from task-specific chest X-ray tasks. Phase 3, chain-of-thought report generation and grounding: 25 hours — findings-first report generation and phrase-to-region grounding. Phase 4, question answering and multi-image handling: 15 hours — multi-turn visual question answering with single-image, multi-image and multi-study inputs. Phase 5, Indian hospital reporting assistant: 15 hours — grounding overlays and plain-language English and Hindi patient summaries over the draft report, demonstrated on illustrated cases. Phase 6, presentation and viva preparation: 15 hours — PPT, demo walkthrough, and the viva Q&A document.

Support

Support covers setup help for the delivered code, a walkthrough of the system and the demo console, and fixes for defects in the delivered code. Guidance is also provided for adapting the report and presentation to the student's college format and for answering guide questions about the base paper, chain-of-thought prompting, grounding, and the student's own contribution. Support does not cover re-running experiments on the student's personal machine beyond the setup walkthrough, and it does not invent results for experiments that were not run.