Crop Yield Forecasting with XGBoost

Crop Yield Forecasting with XGBoost predicts how much a crop will produce per hectare before the season ends, using district, crop, season, seasonal rainfall and mean temperature as inputs. It matters because farmers, procurement agencies and policymakers all make high-stakes decisions — sowing choices, storage planning, MSP procurement — on yield guesses that are often little more than last year's number repeated. Official statistics arrive months late, and simple trend extrapolations collapse in drought or flood years when rainfall deviates sharply from normal. The core difficulty is that yield is tabular, multi-factor and non-linear: the same rainfall helps one district and waterlogs another, temperature interacts with crop stage, and district baselines differ by a factor of three. This project treats yield as a supervised regression problem and learns those interactions from 246,091 district-level records spanning 1997–2015, so the forecast is grounded in measured history rather than heuristics.

Raw APY district statistics and IMD seasonal rainfall are cleaned, merged and shaped into a modeling table: each row is one district-crop-year with features for district, crop, season, lagged district yield, seasonal rainfall, mean temperature, area and crop duration, and the target is yield in tonnes per hectare. The target is log-transformed to tame its right skew, then an XGBoost regressor is trained on 1997–2011 rows — hundreds of shallow trees, each new tree correcting the residual errors of the trees before it, every correction shrunk by the learning rate. Hyperparameters are tuned with grouped 5-fold cross-validation in which districts never leak across folds. The frozen model is evaluated once on the held-out 2014–2015 districts, reporting RMSE, MAE and R². At inference the dashboard encodes a new district-crop-season input into the identical feature vector, inverts the log transform, and returns the point forecast together with its 90% interval.

1. The APY release (246,091 district-crop-year rows, 1997–2015) is cleaned: missing area/production rows dropped, text fields standardized, and yield derived as production ÷ area in tonnes per hectare.
2. IMD seasonal rainfall is merged at district–year grain; mean season temperature, lag-1 district yield, crop duration and area join the feature table. Rainfall and temperature come from the IMD merge — the APY release itself carries only area and production.
3. The yield target is log1p-transformed so that a 10% error on cotton (0.58 t/ha) and on sugarcane (70+ t/ha) are penalized proportionally; the transform is inverted at inference.
4. An XGBoost regressor is trained on 1997–2011 rows with hyperparameters (max_depth, learning rate, tree count, subsample) tuned by grouped 5-fold cross-validation, districts never shared between folds.
5. Quantile heads at the 5th and 95th percentiles are trained alongside the mean head, giving the 90% prediction interval shown on the dashboard.
6. The frozen model is evaluated once on the held-out 2014–2015 districts: RMSE and MAE in t/ha plus R², with predicted-vs-actual and residual plots exported for the report.
7. In the dashboard, a new input is encoded into the identical feature vector, passed through the saved model, and returned as yield (t/ha), interval, and expected production for the entered area.

Tech stack:

• Python 3, XGBoost
• pandas · NumPy
• scikit-learn (grouped splits, metrics)
• Jupyter Notebook (training & evaluation)
• HTML · CSS · vanilla JavaScript (forecasting dashboard)
• Matplotlib (diagnostic plots)
• data.gov.in APY statistics + IMD rainfall
• Git

Dataset & model details

• Dataset: "District-wise, season-wise crop production statistics", Open Government Data Platform India (data.gov.in), Ministry of Agriculture and Farmers Welfare — approximately 246,091 district-crop-year rows spanning 1997–2015, covering 646 districts, 33 states/UTs, 105 crops and 6 seasons. Each row carries state, district, crop year, season, crop, area (hectares) and production (tonnes); yield is derived as production ÷ area. Seasonal rainfall and mean temperature are merged from India Meteorological Department records — they are not in the APY release, and the merge is documented in the notebook.
• Task: Tabular regression; input = feature vector (district, crop, season, lag-1 district yield, season rainfall, mean temperature, area, crop duration); output = predicted yield in tonnes per hectare plus a 90% prediction interval.
• Model: XGBoost regressor (gradient-boosted decision trees). Training minimizes squared error on the log1p-transformed yield; each successive tree fits the residuals of the ensemble so far, with learning-rate shrinkage and row/column subsampling as regularizers. Quantile heads at the 5th/95th percentiles produce the interval.
• Metrics: RMSE and MAE in t/ha and R² on held-out districts (2014–2015). Design targets for the built-to-order training run: RMSE 0.31 t/ha, MAE 0.24 t/ha, R² 0.86. No metric is claimed as measured until the training run is executed for the order; the report records the actual achieved figures.

The build's four strongest capabilities are the trained XGBoost regressor itself, the interval forecast, the grouped evaluation discipline, and the live forecasting dashboard. The regressor learns from 246,091 APY records with IMD rainfall merged in, using a log-transformed yield target so that errors stay proportional across crops as different as cotton at 0.58 t/ha and sugarcane above 70 t/ha. Instead of a single point estimate it returns a 90% prediction interval from quantile heads, so a user sees the honest uncertainty band rather than false precision. Evaluation holds out entire districts — no district appears in both train and test — which measures true generalization to unseen regions instead of memorization. The dashboard wraps all of this in a working web app: pick district, crop, season, rainfall and temperature, and get the forecast with its interval, the district's yield history, and the model's feature-importance breakdown.

• [Trained XGBoost regressor] Gradient-boosted regression on the real APY + IMD feature table: each new tree corrects the residual errors of the trees before it, with the learning rate shrinking every correction so no single tree dominates.
• [90% prediction intervals] Quantile regression heads (5th and 95th percentiles) turn every forecast into a range, so the dashboard shows uncertainty honestly instead of a spuriously exact number.
• [Live forecasting dashboard] Enter district, crop, season, rainfall, temperature and area; the app returns predicted yield (t/ha), the interval, and expected production in tonnes, with the district's yield trajectory charted.
• [District-grouped evaluation] Train/validation/test splits are grouped by district — no district leaks across splits — so reported metrics reflect performance on genuinely unseen regions.
• [Log-target training recipe] The yield target is log1p-transformed before training (yields are right-skewed) and inverted at inference, which keeps errors proportional across low- and high-yield crops.
• [Feature importance & diagnostics] Gain-based importance ranking, predicted-vs-actual scatter and residual plots, all exported for the report and the viva.
• [Historical trend explorer] Eighteen years of district yield history with seasonal rainfall overlaid, showing exactly the drought-year dips the rainfall feature must explain.
• [Reproducible training notebook] Data cleaning, APY–IMD merge, feature engineering, tuning and evaluation in one notebook with fixed seeds.

• Complete training & evaluation Jupyter notebook (cleaning, APY–IMD merge, features, tuning, evaluation)
• Trained XGBoost model file with the preprocessing pipeline
• Forecasting web dashboard (the working demo, wired to the model interface)
• Evaluation pack: RMSE/MAE/R², predicted-vs-actual scatter, residual plots, feature-importance chart
• Project report PDF (background, gradient-boosting theory, methodology, results, limitations)
• PPT presentation for final review
• Viva Q&A preparation document (boosting vs bagging, residuals, log transform, leakage, quantile intervals)