Federated LoRA fine-tuning of LLMs — IEEE Transactions 2025 Base Paper Project

Federated LoRA fine-tuning of LLMs answers a practical problem: several organisations want one better language model but cannot pool their text because it is private, regulated, or simply too large to move. Full fine-tuning makes this worse, because every client must upload a complete copy of a large model in every round, which costs bandwidth, time, and hardware that smaller participants do not have. Low-Rank Adaptation (LoRA) freezes the base model and trains only two small matrices per adapted layer, so each update is a fraction of the full model size. Federated learning adds the second half: every client trains its adapters on its own data, and only the adapters travel to the server for aggregation. This project implements that combination on a small open language model, following the ECLoRA base paper, and adds the student's own evaluation: heterogeneous client ranks, a communication-cost comparison, and a rank-ablation study.

Note on terminology: LoRA here means Low-Rank Adaptation, a parameter-efficient fine-tuning method. It is not LoRa radio, the wireless technology used in several other catalogue listings.

This is a built-to-order software project: the federated training harness, experiment report, dashboard demo and the complete viva kit (report, PPT, Q&A) are prepared fresh for you. Starting price: Rs 30,000 (final price confirmed after topic, base paper and compute scope are locked).

Base paper at a glance

Base paper

• Paper: "Federated Fine-Tuning on Heterogeneous LoRAs With Error-Compensated Aggregation" (ECLoRA), Ning et al., IEEE Transactions on Neural Networks and Learning Systems, 2025.
• DOI: https://doi.org/10.1109/TNNLS.2025.3586545
• Problem the paper tackles: in federated fine-tuning, clients have different hardware, so forcing one LoRA rank on all of them creates a "bucket effect" where the weakest client limits everyone. Earlier heterogeneous-rank aggregation used zero-padding, stacking, or classical SVD, which the paper reports as losing precision or adding heavy overhead.
• What is implemented from the paper: local LoRA training on each client with a client-specific rank; adapter-only upload to the server; aggregation of heterogeneous adapters with randomized SVD (RSVD) instead of zero-padding; and error compensation, where the decomposition error from one round is carried into the next round to improve aggregation precision.
• Paper-reported results (the paper's own findings, not this build's measured results): the paper reports faster convergence and substantially lower aggregation time than classical SVD. This build does not claim those findings as its own; its report documents whatever the student's own experiment runs actually produce.

Own features

These are the student's own contribution, clearly distinct from the base paper:

• [Communication-cost comparison] A dashboard and report table comparing full fine-tuning against LoRA at ranks 4, 8, 16 and 32, computed from parameter counts for the chosen base model, so the saving is shown in megabytes per client per round rather than asserted.
• [Heterogeneous-client experiment bench] Five simulated clients with different ranks, dataset sizes and non-IID data skews (code instructions, general QA, summarisation), so the viva can discuss what happens when clients are unequal, not just identical.
• [Rank-ablation study] The same federated run repeated at several ranks, reported as a trade-off table: adapter size and upload cost against held-out loss, with the rank choice justified from the student's own numbers.
• [Error-compensation residual tracker] A per-round record of the carried-over decomposition error, showing whether compensation is shrinking the residual across rounds.
• [Guide-ready experiment report] Every experiment above is written up with its configuration, procedure and observed outcome, so a guide can verify the work without re-running code blind.

Federated LoRA fine-tuning works as a repeated loop between one server and several clients, and this project implements each stage so the student can explain it. The server starts by sending the frozen base model and an initial adapter configuration to every client. Each client then trains only its own LoRA adapters on its private local text for a few local epochs, keeping its raw data on its own machine throughout. Clients upload just their trained adapters, choosing a rank their hardware can handle, and the server aggregates these unequal adapters with randomized SVD while carrying forward the previous round's decomposition error as compensation. The merged global adapter is sent back to the clients, and the loop repeats for the configured number of rounds. After training, the global adapter is evaluated on a held-out set, and the rank-ablation runs repeat the loop at different ranks so the final report can compare cost against quality on the student's own numbers.

1. The server initialises the global LoRA adapter and distributes the frozen base model plus adapter configuration to all clients.
2. Each client trains only its LoRA adapters on its own local instruction data for a fixed number of local epochs; base-model weights are never updated and raw text never leaves the client.
3. Each client uploads only its trained adapter, at its own rank (rank 4 to rank 16 in the reference bench), keeping uploads small.
4. The server aggregates the heterogeneous adapters with randomized SVD and applies error compensation using the decomposition error carried from the previous round.
5. The merged global adapter is broadcast back to the clients, and steps 2–4 repeat for the configured number of aggregation rounds (12 rounds in the reference run, design target).
6. After the final round, the global adapter is evaluated on a held-out instruction set: loss and perplexity are recorded, never asserted in advance.
7. The loop is repeated at ranks 4, 8, 16 and 32 for the rank-ablation study, and all results are written into the experiment report with their configurations.

Tech stack:

• Python 3 · PyTorch
• Hugging Face Transformers
• Hugging Face PEFT · LoRA
• GPT-2 small · 124M parameters
• Stanford Alpaca instruction subset
• HTML · CSS · JavaScript dashboard
• Matplotlib · experiment plots
• Git · Jupyter Notebook

Dataset & model details

• Dataset: Stanford Alpaca instruction dataset (52,000 instruction-following examples, public release) — a curated 2,000-example subset is used for this build (design target), partitioned non-IID across five clients by task category (code instructions, general QA, summarisation), with 320 examples held out for evaluation (design target).
• Task: Instruction fine-tuning of a causal language model; input = an instruction prompt, output = a response text generated by the adapted model.
• Model: GPT-2 small (124M parameters, open weights) with LoRA adapters injected into the attention projections; base weights frozen, only adapter matrices A and B trained. Reference ranks: 4, 8, 16 and 32 for the ablation study; heterogeneous client ranks 4, 8 and 16 in the federated bench.
• Metrics: Training and validation loss, held-out perplexity, adapter parameter count, upload megabytes per client per round, and rounds run. No accuracy or quality figure is claimed before the student's own runs; the report records the figures those runs actually produce.

Federated LoRA fine-tuning of LLMs delivers five working capabilities: adapter-only federated training, heterogeneous client ranks, error-compensated aggregation, a live cost comparison, and an experiment report a guide can audit. The training harness freezes the base model and trains only LoRA adapters on each client, so raw text never leaves a client machine and each upload is a small adapter file rather than a full model. Clients choose ranks that fit their own hardware, from rank 4 on a weak node to rank 16 on a lab GPU, and the server aggregates these unequal adapters with randomized SVD plus error compensation from the previous round. A dashboard makes the whole loop visible: per-client loss, global loss across rounds, and the communication saving against full fine-tuning in megabytes. Finally, the rank-ablation study turns the build into a defensible experiment, because the report compares ranks on the same data and states which rank the student recommends and why.

• [Adapter-only federated training] Each client trains LoRA adapters on its own local text while base-model weights stay frozen; only adapter weights are uploaded, so private text never leaves the client.
• [Heterogeneous client ranks] Clients run different LoRA ranks matched to their hardware (rank 4 to rank 16 in the reference bench), instead of forcing every client onto the weakest client's configuration.
• [Error-compensated RSVD aggregation] The server aggregates unequal adapters with randomized SVD and carries the decomposition error into the next round, following the ECLoRA method rather than zero-padding adapters to a common size.
• [Communication-cost dashboard] A working web console shows per-client upload size, cumulative upload across rounds, and a full fine-tuning versus LoRA comparison computed from parameter counts.
• [Rank-ablation experiment] The same federated run is repeated at ranks 4, 8, 16 and 32 (design target set) and reported as a trade-off table of adapter size, upload cost and held-out loss.
• [Non-IID client bench] Five clients hold differently skewed data (code instructions, general QA, summarisation), so the evaluation reflects unequal real-world clients rather than identical copies.
• [Complete viva kit] Source code, experiment report, presentation and viva Q&A document are delivered together, with every claimed figure traced to a configuration or a run the student can explain.

• Complete federated LoRA training harness source code (client training, RSVD aggregation, error compensation)
• Communication-cost dashboard demo (the web console shown on this page)
• Rank-ablation and heterogeneous-client experiment report with configurations and observed outcomes
• Project report PDF (background, base paper, methodology, experiments, limitations)
• PPT presentation for final review
• Viva Q&A preparation document (federated learning, LoRA, aggregation, heterogeneity, communication cost)
• Starting price: Rs 30,000 (final price confirmed after topic, base paper and compute scope are locked)

Work plan & timeline

Total work plan: 120 hours, split across six phases. Phase 1, base-paper study and environment setup: 15 hours — reading the ECLoRA paper, setting up Python, PyTorch and the PEFT stack, and reproducing the LoRA adapter arithmetic. Phase 2, data preparation and federated harness: 25 hours — curating the instruction subset, building non-IID client partitions, and implementing local client training with adapter-only upload. Phase 3, ECLoRA aggregation: 25 hours — implementing randomized SVD aggregation of heterogeneous adapters plus the error-compensation mechanism. Phase 4, experiments: 20 hours — running the rank-ablation study and the heterogeneous-client bench and recording cost and loss outcomes. Phase 5, evaluation and report: 20 hours — held-out evaluation, cost tables, diagrams and the full project report. Phase 6, presentation and viva preparation: 15 hours — PPT, demo walkthrough and the viva Q&A document.

Support

Support covers setup help for the delivered code, a walkthrough of the training harness and the experiment report, and fixes for defects in the delivered code. Guidance is also provided for adapting the report and presentation to the student's college format and for answering guide questions about the base paper, the aggregation method, and the student's own contribution. Support does not cover re-running experiments on the student's personal machine beyond the setup walkthrough, and it does not invent results for experiments that were not run.