Building a palm oil news pipeline from LSEG

A palm oil desk gets hundreds of headlines a day: Malaysian export estimates, Indonesian levy changes, Indian import duties, biodiesel mandates and weather. Only a handful move the market. This is how I built a small pipeline that finds those, stores them, and delivers them without anyone scrolling a terminal.

1. Extract: the LSEG Data Library

The lseg.data Python library opens a session to LSEG Workspace and queries headlines and story bodies by search terms and date range. Story bodies come back as HTML, so BeautifulSoup strips them to clean text.

import lseg.data as ld
ld.open_session()
heads = ld.news.get_headlines("palm oil OR CPO OR FCPO", count=100)

2. Filter: rules before models

I started with keyword rules rather than machine learning. They are easy to explain and easy to fix when they go wrong:

• Include stories that name the product with a price or trade word: physical, FOB, CIF, export, levy.
• Exclude noise such as equity stories about plantation companies or generic market wraps.
• Deduplicate by story ID. Agencies often update the same story several times a day.

3. Load: Parquet, not CSV

Filtered stories go into Parquet files partitioned by date. Parquet keeps the data types, compresses news text well, and loads in a fraction of the time CSV takes once you have months of history. It also feeds straight into pandas or a BI tool for counting topics over time.

4. Deliver: a digest people actually read

The daily email is plain HTML, one line per story with the time, source and headline, sent over SMTP. A small JSON state file records which story IDs have already been sent, so a re-run never sends duplicates. That one detail made the job safe to schedule every few minutes.

What made it reliable

• Idempotence: the state file means running twice does no harm.
• Credentials outside the code: email and API secrets come from environment variables, never from the repository.
• Fail loudly, recover quietly: a failed LSEG session is logged and retried on the next run instead of crashing the scheduler.
• Small pieces: extract, filter, store and send are separate functions, so each can be tested and changed alone.