Crop Price Forecasting using XGBoost and Historical Mandi Data

Farmers and traders lose money when they sell at the wrong time: onion prices can double in a fortnight or crash by half after a bumper arrival, yet most mandi decisions are still made on word-of-mouth and last week's rate. Meanwhile the government publishes daily min, max and modal prices for hundreds of mandis on Agmarknet and eNAM — a rich public time series that most student projects ignore. This project turns that raw mandi data into a genuine forecasting system: weekly modal-price series are cleaned and feature-engineered (lags, rolling means, arrival volumes, seasonality), an XGBoost regressor is trained with walk-forward validation, and the result is served through an interactive dashboard where the student picks a commodity and mandi and sees the historical curve plus an 8-week forecast with an uncertainty band. The built-to-order training run is executed on the buyer's chosen commodity–mandi pair, and the report documents the real measured error from the walk-forward folds — no invented accuracy anywhere.

1. Daily mandi records (commodity, mandi, min/max/modal price, arrivals) are pulled from the Agmarknet/eNAM public portals and loaded into the notebook.
2. Records are deduplicated, missing market days are handled, and the series is resampled to a weekly modal-price series per commodity–mandi pair.
3. Features are engineered per week: lags at 1/2/4/8 weeks, 4-week and 8-week rolling means and volatility, arrival volume, month and season flags, and a rainfall-anomaly indicator where available.
4. An XGBoost regressor (300 estimators, max depth 5, learning rate 0.05, subsample 0.8) is trained on the feature matrix with walk-forward splits: each fold trains on history and tests on the next 8 unseen weeks.
5. MAPE, RMSE and directional accuracy are computed per fold; feature importances and the worst-fold cases are logged for the report.
6. The dashboard loads the trained model artifacts: selecting a commodity, mandi and horizon renders history plus the forecast path with its widening uncertainty band.

Tech stack:

• Python 3, XGBoost, scikit-learn
• pandas, NumPy (data pipeline)
• Matplotlib (evaluation plots)
• Jupyter Notebook (training & evaluation)
• HTML5 + JavaScript dashboard (canvas charts, single-file)
• Agmarknet / eNAM public mandi price records

Dataset & model details

• Dataset: Agmarknet (agmarknet.gov.in) and eNAM public mandi price records — daily commodity-wise, mandi-wise min/max/modal prices (₹/quintal) plus arrival volumes. The reference configuration covers commodities such as wheat, paddy, soybean, onion, cotton and turmeric across representative mandis (e.g. Lasalgaon/Nashik, Indore, Nagpur, Jaipur, Bengaluru, Erode), resampled to weekly modal-price series of 52+ weeks per commodity–mandi pair.
• Task: Regression — predict the weekly modal price N weeks ahead (4/8/12-week horizons); input = engineered feature vector for week t, output = modal price at week t+N.
• Model: XGBoost regressor — reference configuration 300 estimators, max_depth 5, learning_rate 0.05, subsample 0.8, colsample_bytree 0.8; walk-forward validation with expanding window and 8-week test folds.
• Metrics: MAPE (design target ≤ 12% on walk-forward folds), directional accuracy (design target ≥ 60%), RMSE reported per commodity–mandi. No accuracy is claimed as measured until the built-to-order training run executes — the report documents the actual achieved figures.

• [XGBoost forecasting model] Gradient-boosted tree regressor trained on engineered mandi features (lags, rolling statistics, arrivals, seasonality) with a documented hyperparameter set (300 trees, depth 5, lr 0.05).
• [Interactive forecast dashboard] Pick commodity, mandi and horizon (4/8/12 weeks); see history, the forecast path and an uncertainty band, plus current price and expected direction cards.
• [Agmarknet/eNAM data pipeline] Cleaning and resampling pipeline for public mandi records: dedupe, missing-day handling and weekly modal-price series construction, all in the notebook.
• [Walk-forward evaluation] Time-series-correct backtesting (expanding training window, 8-week test folds) with MAPE, RMSE and directional-accuracy reporting — the right way to evaluate a forecaster.
• [Feature importance] XGBoost gain-based importance table showing which signals drive the forecast (typically lag-1 and rolling means), included in the report.
• [Uncertainty bands] Forecast intervals widening with horizon, giving the student an honest way to discuss confidence in the viva.
• [Full training notebook] Data loading, feature engineering, model training, hyperparameter notes, evaluation and forecast export in one reproducible notebook.

• Data cleaning & resampling notebook for Agmarknet/eNAM records
• Feature-engineering and XGBoost training notebook with walk-forward evaluation
• Trained model artifacts for the chosen commodity–mandi pair(s)
• Interactive forecast dashboard (commodity/mandi/horizon, uncertainty bands)
• Evaluation plots: walk-forward error, feature importance, forecast-vs-actual
• Project report PDF (background, data pipeline, model theory, methodology, measured results)
• PPT presentation for final review
• Viva Q&A preparation document (XGBoost, time-series validation, MAPE vs RMSE, data leakage)