3UR-LLM End-to-End Multimodal 3D Scene Understanding — IEEE Transactions 2025 Base Paper Project

Asking a language model about a room sounds simple until the room is three-dimensional. A photograph flattens depth, hides whatever sits behind the sofa, and changes completely when the camera moves. A point cloud does not have those problems: it records the room's geometry directly, point by point. The 3UR-LLM paper, published by Xiong, Zhuge, Zhu, Zhang and Lu in IEEE Transactions on Multimedia in 2025, builds an end-to-end multimodal large language model around exactly that input. The model receives a 3D point cloud directly, fuses its features with the text instructions, and projects them through a 3D compressor module into a manageable set of tokens for the language model. A data pipeline built from open-source 2D MLLMs and LLMs generates high-quality 3D-text pairs, forming the 3DS-160K pre-training set, and the resulting model supports 3D scene description, dense captioning, and question answering. This project implements that model for a student build, evaluated on the public ScanQA and SQA3D benchmarks, and adds a clearly separate Indian contribution: indoor scenes from Indian homes and classrooms captured with a phone RGB-D sensor, bilingual English/Hindi scene question answering, and an interactive scene-QA demo console with a visible token-compression panel.

This is a built-to-order software project: the 3D scene-understanding model, demo console, report, presentation, and viva Q&A are prepared fresh for you. Starting price: Rs 29,000 (final price confirmed after topic, base paper and compute scope are locked).

The Problem

Language models understand text, and vision-language models understand pictures, but a room is neither: it is a volume with objects arranged in space, distances between them, and parts hidden behind other parts. Pipelines that start from 2D images have to reconstruct that volume indirectly, and every reconstruction step adds error — a chair seen from one angle is a different shape from another, and an object fully occluded in the photograph simply does not exist for the model. Separate conversion stages also make the system harder to explain: when an answer is wrong, it is unclear whether the image encoder, the 2D-to-3D bridge, or the language model is at fault. The 3UR-LLM paper addresses this by removing the bridge: the point cloud goes straight into the model, a 3D compressor module compresses the spatial cues and textual narrative into a manageable token set, and the language model reasons over those tokens directly. The student build reproduces that reasoning chain on public benchmarks, then faces the variant the paper does not cover: Indian indoor scenes, where room layouts, furniture, and everyday objects differ from the paper's ScanNet-style setting, and where a student demonstrating the system may need to ask questions in Hindi as well as English.

Base paper at a glance

Base paper

• Paper: "3UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding", Haomiao Xiong, Yunzhi Zhuge, Jiawen Zhu, Lu Zhang and Huchuan Lu, IEEE Transactions on Multimedia, 2025, vol. 27, pp. 2899–2911.
• DOI: https://doi.org/10.1109/TMM.2025.3557620 (IEEE Xplore document 10981614). Free preprint consulted separately: https://arxiv.org/pdf/2501.07819 — the preprint is not the DOI record and is not conflated with it here.
• Problem the paper tackles: 3D scene understanding with a language model is hard because existing approaches do not take the 3D scene directly as input; they rely on 2D views or conversion stages that lose spatial structure, and they lack large-scale high-quality 3D-text training pairs.
• What is implemented from the paper: the end-to-end 3D MLLM that directly receives a 3D point cloud as input; the 3D compressor module that compresses 3D spatial cues and textual narrative; projection of 3D features fused with text instructions into a manageable set of LLM tokens; the data-pipeline idea of generating high-quality 3D-text pairs from open-source 2D MLLMs and LLMs; and the three supported tasks — 3D scene description, dense captioning, and question answering — evaluated on the public ScanQA and SQA3D benchmarks.
• What is deliberately not claimed: no result beyond the paper's own headline figure is quoted, because no other number from the paper is verified for this listing; the 7.1% CIDEr figure is quoted only as the authors' reported figure, never as a promise for the student build; and ScanNet-style data is named only as the paper's setting, never as the student build's dataset.
• Paper-reported results (the paper's own findings, not this build's measured results): the authors report that 3UR-LLM exceeds its counterparts by 7.1% CIDEr on ScanQA, while utilizing fewer training resources. No other accuracy, latency, or metric number is attached to that finding here, and the student build's report records only the figures its own ScanQA and SQA3D runs actually produce.

Own features

These are the student's own contribution, clearly distinct from the base paper:

• [Indian indoor scenes from a phone RGB-D sensor] The paper's setting is ScanNet-style indoor scans. The adaptation layer captures Indian homes and classrooms with a phone RGB-D sensor instead, so the demo scenarios cover room layouts, furniture, and everyday objects the paper's setting does not represent. This is presented as a demonstrated scenario layer, not as a validated model on Indian data, and no Indian dataset is claimed.
• [Bilingual English/Hindi scene question answering] The same scene-QA console accepts questions in English and in Hindi, so a student can demonstrate the system to a reviewer in either language. The bilingual layer is the team's addition: the paper's model is not described as bilingual.
• [Interactive scene-QA demo console with a visible token-compression panel] A demo that shows the room as a labelled scene, answers questions from a visible scene graph, and displays the token-compression step — how many point-cloud tokens the compressor reduces to — so a reviewer can see the paper's core mechanism working instead of taking it on trust.

The pipeline runs as a visible chain, and each stage's output is shown in the demo so a student can explain it in a viva. First, a 3D point cloud of an indoor scene is loaded — in the student build's evaluation, the public ScanQA and SQA3D benchmarks supply the scenes and questions, while the demo console shows an Indian home or classroom scene captured with a phone RGB-D sensor. The point-cloud encoder turns the cloud into 3D features. The 3D compressor module compresses the spatial cues and the textual narrative into a manageable token set, and the compression panel shows that reduction happening. The fused 3D features and text instructions are projected into LLM tokens, and the language model produces its output for the selected task: a scene description, dense captions for the objects, or an answer to a question. The bilingual layer accepts the question in English or Hindi, and the console keeps the paper's mechanism and the team's adaptation visibly separate.

1. Load a 3D point cloud of an indoor scene (ScanQA and SQA3D benchmark scenes for evaluation; a phone RGB-D capture of an Indian home or classroom for the demo console).
2. Encode the point cloud into 3D features with the point-cloud encoder.
3. Run the 3D compressor module: compress the 3D spatial cues and textual narrative into a manageable set of tokens, and show the reduction in the token-compression panel.
4. Project the fused 3D features and text instructions into LLM tokens.
5. Generate a scene description: room type, main objects, and their arrangement, in plain language.
6. Generate dense captions: one caption per object, with attributes and positions.
7. Answer scene questions: match the question to the scene graph and produce a grounded answer in English or Hindi.
8. Evaluate on the public ScanQA and SQA3D benchmarks, and record only the figures the student's own runs produce.
9. Show the Indian adaptation panel: phone RGB-D capture, Indian home and classroom scenarios, and bilingual question answering, kept distinct from the paper's ScanNet-style setting.
10. Display every stage — point cloud, compressed tokens, description, captions, and answers — in the demo console for review and viva demonstration.

Tech stack:

• Python 3 · PyTorch · Hugging Face Transformers
• Point-cloud encoder · 3D compressor module
• ScanQA public benchmark · SQA3D public benchmark
• HTML · CSS · JavaScript demo console
• Open3D · point-cloud visualisation
• Matplotlib · evaluation plots
• Pandas · NumPy data handling
• Git · Jupyter Notebook

Dataset & model details

• Dataset (paper): 3DS-160K, 160,000 high-quality 3D-text pairs generated by a data pipeline from open-source 2D MLLMs and LLMs, used for pre-training. The paper's setting is ScanNet-style indoor 3D scenes.
• Dataset (student build): The public ScanQA and SQA3D benchmarks for evaluation of scene description, dense captioning, and question answering. The build does not claim the paper's 3DS-160K pre-training data as its own dataset, and its results are not directly comparable to the paper's findings.
• Dataset (Indian adaptation): Indian home and classroom scenes captured with a phone RGB-D sensor for the demo console. The adaptation is demonstrated on illustrated demo scenarios in the console, not on a published Indian dataset, and it is labelled as such wherever it appears. No Indian dataset is claimed.
• Task: 3D scene understanding; input = a 3D point cloud plus a text instruction, output = a scene description, dense object captions, or an answer to a scene question.
• Model: 3UR-LLM as described in the paper — an end-to-end 3D MLLM with direct point-cloud input, a 3D compressor module, and projection of fused 3D features and text instructions into LLM tokens. No architecture detail beyond the paper's description is claimed for it here.
• Metrics: CIDEr for captioning-style evaluation, as reported by the paper on ScanQA; the student build's report records only the figures its own ScanQA and SQA3D runs actually produce, and no figure is claimed before those runs.

3UR-LLM scene understanding delivers seven working capabilities: direct point-cloud input, a 3D compressor, LLM token projection, scene description, dense captioning, scene question answering, and a bilingual Indian-scene demo console, all visible in a demo a reviewer can operate. The point cloud enters the model directly, so no 2D-to-3D conversion stage stands between the room and the answer. The compressor shrinks the cloud's spatial cues into a manageable token set, which keeps the language model's context usable instead of flooding it with raw points. Token projection fuses those compressed 3D features with the text instructions, so the same model can describe a scene, caption its objects densely, and answer questions about it. The three own-feature layers then make the system demonstrable in an Indian setting: rooms captured on a phone, questions asked in English or Hindi, and a console that shows the compression step happening.

• [Direct point-cloud input] Receives a 3D point cloud of the scene directly as model input, so the model reasons over true room geometry instead of reconstructed 2D views.
• [3D compressor module] Compresses 3D spatial cues and textual narrative into a manageable set of tokens, keeping the language model's context usable instead of flooding it with raw points.
• [LLM token projection] Projects the 3D features fused with text instructions into LLM tokens, so one model handles description, captioning, and question answering.
• [3D scene description] Produces a plain-language description of the whole scene — the room type, its main objects, and how they are arranged — from the point cloud.
• [Dense captioning] Captions individual objects in the scene with their attributes and positions, rather than describing the room only as a whole.
• [Scene question answering] Answers questions about the scene — what is present, where it is, and how objects relate — grounded in the scene's 3D structure.
• [Bilingual Indian-scene demo console] Shows Indian home and classroom scenes captured with a phone RGB-D sensor, accepts English and Hindi questions, and displays the token-compression panel, keeping the team's contribution visibly separate from the paper.

• Complete 3UR-LLM scene-understanding pipeline source code (point-cloud encoder, 3D compressor, token projection, description, dense captioning, question answering, Indian adaptation layer, bilingual layer)
• Interactive demo console (the web demo shown on this page) with scene view, description tab, dense-captioning tab, question-answering tab, and token-compression panel
• Project report PDF (background, base paper, methodology, experiments, limitations)
• PPT presentation for final review
• Viva Q&A preparation document (3UR-LLM, 3D compressor, token projection, ScanQA vs SQA3D, Indian adaptation, bilingual QA, limitations)
• Starting price: Rs 29,000 (final price confirmed after topic, base paper and compute scope are locked)
• Specific next step: message on WhatsApp for a quotation stating scope (3UR-LLM 3D scene understanding plus Indian indoor-scene adaptation plus bilingual English/Hindi scene QA), starting price Rs 29,000, and delivery under the 110-hour work plan as a built-to-order project

Work plan & timeline

Total work plan: 110 hours, split across six phases. Phase 1, base-paper study and environment setup: 15 hours — reading the 3UR-LLM paper, setting up Python, PyTorch, and the ScanQA and SQA3D data pipeline. Phase 2, point-cloud encoder and compressor: 25 hours — point-cloud encoding, the 3D compressor module, and token projection into the language model. Phase 3, tasks and evaluation: 25 hours — scene description, dense captioning, and question answering, with evaluation on ScanQA and SQA3D. Phase 4, Indian adaptation and bilingual layer: 15 hours — phone RGB-D capture scenarios for Indian homes and classrooms, and English/Hindi question answering. Phase 5, evaluation and report: 15 hours — benchmark evaluation, diagrams, and the full project report. Phase 6, presentation and viva preparation: 15 hours — PPT, demo walkthrough, and the viva Q&A document.

Support

Support covers setup help for the delivered code, a walkthrough of the model and the demo console, and fixes for defects in the delivered code. Guidance is also provided for adapting the report and presentation to the student's college format and for answering guide questions about the base paper, the 3D compressor, and the student's own contribution. Support does not cover re-running experiments on the student's personal machine beyond the setup walkthrough, and it does not invent results for experiments that were not run.