Case study: an MPOB nowcast that keeps score
The Malaysian Palm Oil Board's monthly report moves palm oil prices on the day it comes out. My MPOB Nowcast estimates the report before it is published. This note covers the three things that make it worth trusting: the data arrives on its own, every forecast is frozen before the release, and the model only changes when a back-test says it should.
1. Getting the data without typing it
The first version relied on me entering the figures each month. That is slow, and it hides mistakes. MPOB keeps only its latest report online, as an HTML table inside an embedded frame, so a script now follows the chain: release list → newest article → embedded report, and reads four lines: crude palm oil production, total palm oil stocks, exports and imports.
- Revisions are applied. Each report also revises the previous month, so the script corrects that row before adding the new one.
- Domestic use is the balance residual: opening stocks + production + imports − exports − closing stocks.
- It refuses to guess. A missing month, a negative residual or a figure outside its usual range stops the job and emails me, because those usually mean the page layout changed.
The first run paid for itself: my hand-entered September figures were slightly off (exports 1,120,395 t against MPOB's 1,097,074 t). Then, on the first release day, GitHub skipped the scheduled run altogether. The job now tries four times on each release day; once the month is in, the extra runs do nothing.
2. Freezing forecasts before the release
A back-test can be quietly re-run until it looks good. A live record cannot. When MPOB publishes a month, the next month's forecast is written to a log file and never edited again. The file's git history shows when each forecast was saved, so anyone can check it came before the release. The page then scores each forecast once MPOB reports: the miss, whether the direction of the stock change was right, and whether the actual landed inside the stated range.
The range is the 10th to 90th percentile of the model's past misses over 36 back-tested months, not plus or minus a standard error. Past misses lean one way at times, and the range should show that.
The first entry was October 2026. The forecast was on the site at 11:49 on release day, 41 minutes before MPOB published. It said stocks of 3,748k t; MPOB reported 3,621k t. The direction (a build) was right and the actual was inside the range, but the size was 126k t too high, mostly because exports came in 156k t above the estimate.
3. Changing the model only on evidence
One thing in the October estimate looked wrong to the eye: exports were forecast to jump 455k t in a month, far outside anything seen for September to October since 2014. MPOB's actual number was higher still. Intuition about a single month is a poor guide.
November was a better reason to look. The original model took last year's figure for the same month and scaled it by the recent trend. November 2025 had the biggest export drop on record, so the model copied it into November 2026 and forecast a 610k t stock build to 4.23 Mt. That would need the lowest November exports in the data.
So I back-tested alternatives on every month from 2020 to October 2026, each using only data available before that month:
| Formula | Average stock miss | Direction right | Nov 2026 stocks |
|---|---|---|---|
| Original: last year's month × recent trend | 188k t | 46 of 82 | 4,231k t |
| Seasonal: last month × usual seasonal change | 174k t | 55 of 82 | 3,737k t |
| Chosen: seasonal production; exports half each | 167k t | 53 of 82 | 3,960k t |
The chosen formula had the smallest misses in every test window and nearly the best direction record. It starts from where the market actually is (last month's figure) and applies the normal seasonal move, averaged over five years so that one odd year cannot dominate. Exports keep half of the original rule because they are choppy.
Two honest caveats. First, I tried five variants on the same history and kept the best, so the 167k t flatters it a little. Second, it would not have beaten the original in October: it would have said 4,006k t. October's forecast stays in the record exactly as it was made, labelled with the old formula.
What I took from it
- Automate the boring part first. The scraper found a data error on day one.
- Make the record impossible to fudge. Freezing forecasts in git costs nothing and changes how much a reader can trust the page.
- Test, don't argue. Both my October hunch and the November model looked plausible; only the back-test separated them.
- Report the uncertainty. A typical miss of about 160–190k t is large next to a normal monthly move. The range says so plainly.
The real test is the live track record over the next 6–12 months. If the new formula does worse live than in the back-test, the page will show it.
Built with: Python (requests, a small HTML parser, about 30 automated tests), GitHub Actions for the monthly fetch, a static site on GitHub Pages, and hand-written JavaScript charts. Code: fetch_mpob.py, nowcast.py, the page.
For education only. Not trading advice.