Q&A:
What is "Greenness"?
Greenness is the percentage of electricity that comes from renewables. 100% means all electricity generation for the region (state) is coming from renewable sources; 0% means none.
Why is this useful?
You could use this data to reduce your carbon footprint, learn more about electricity/renewables and - in some cases - reduce energy bills.
Reduce your carbon footprint:
If you use electricity when Greenness is high, your carbon footprint is lower. This simple-sounding claim is surprisingly complex; read below for details. Nevertheless, here are some ideas that can reduce your footprint:
- Implement smart heating/cooling that drives harder during high Greenness
- Choose when to charge batteries or other storage systems
- Choose when to charge electric vehicles
- Schedule electric hot water systems, washers and dryers
- Arrange when energy-intensive workloads at your workplace are schedulled
- [Your ideas here!]
Can you use this data to reduce your energy bills?
Unfortunately, everyday households can't yet take advantage of this data to lower bills. There are some retailers that offer pass-through pricing for electricity but since the 2022 energy crisis they are significantly more expensive than standard options.
Some large commercial/industrial customers are exposed to wholesale price fluctuations. It's plausible those customers could save by scheduling loads during predicted low prices
Where do the forecasts come from?
Every 2 hours, GreenForecast.au servers grab data from AEMO?the Australian Energy Market Operator, who run the "National" Electricity Market, a weather forecast and two other minor sources. This data is fed into a custom Machine Learning (ML) model which has been trained on 10 years (roughly a million data points) of data. The current model is an "ensemble" of around a thousand specialised sub-models which predict price and greenness on a 7 day horizon for 5 NEM regions (states).
For technical details, see below.
How accurate is it?
Greenness is "a little better than a weather forecast"; price is a little worse. Unsurprisingly, accuracy for tomorrow is better (about 2x) than 7 days away. On a held-out validation set of data, the model predicts Greenness with an average error of less than 5% and price with average error of 2¢/kWh, depending on region.
What Feedback would you like to hear?
First, thanks for having a look! Any and all feedback greatly appreciated, through any medium you might find convenient. Here are some prompters:
- What was confusing at first?
- How do you think this data could usefully be used?
- Can the data be presented better?
- Energy people: is "greenness" as defined here the right measure?
- Energy people: Do you agree with the rationale in the "Is your carbon footprint really lowered by using at times of high Greenness?" question below?
- ML People: Any suggestions for alternative architectures or tricks to try?
- How can these forecasts be shared with people who might benefit?
Q&A From Energy People:
Greenness Details:
For simplicity, the analysis assumes fuel sources are either 'Green' or 'Not Green'. Wind, solar and hydro are 'Green' while coal, diesel and gas (in various forms) are not. Reality is not so simple but these complexities don't currently substantially affect the overall result.
Batteries and pumped storage will be more important in coming years but are currently small. The model currently assumes they do not affect the level of greenness when discharging (or charging). A future improvement could be to assume they discharge with the same level of Greenness that they were charged with (typically high, because it is cheap to charge when solar is abundant). Currently, this effect would be too small to notice.
Is your carbon footprint really lowered by using at times of high Greenness?
It's Complicated, but yes.
Currently, the vast majority of electricity that is generated is "used" in less than a second. This will remain true until we have (HEAPS) more battery and pumped storage. It makes intuitive sense that the Greenness of your electricity is calculated from whatever power plants are generating right now.
But if someone moves their consumption from a 'dirty' time to a 'clean' time, does that cause the total production of green energy to increase? In the short term, generally not?Wind, solar and hydro typically generate regardless of demand; if usage increases or decreases, they rarely change output, because their fuel is free. (Hydro does respond to demand changes in the short-term, but ultimately the total amount of water available is non-negotiable). This leaves only gas that responds to changes in demand, which means moving consumption from one time to another simply changes when some gas turbine runs, not how much.
Now, there is an exception to this: when there is very high Greenness, excess supply (high wind and sun) the price goes negative. Here, solar and wind plants will reduce their output to avoid excess supply causing problems in the grid. Moving consumption to these times would indeed enable higher renewable generation. Currently, this happens only on occasional days in SA during spring/summer, but will increase in prevalence in coming years.: more people share the same amount of green energy. But in the long term, moving our consumption to times of high Greenness does enable more green power plants to be built: money that would otherwise be needed for expensive batteries (or rapid-response gas turbines) is spent on cheap wind and solar. Indeed, there are already areas where it is less profitable to add wind/solar because of excess capacity.
Put another way, if everybody magically decided to schedule their big loads (eg heating/cooling, EV charging) during times of high Greenness, our transition to 100% renewables would be faster and our electricity bills cheaper.
What about WA? NT?
Sorry, this system is designed around NEM data, which only covers 5 states (and ACT folded in with NSW). If you're a machine learning engineer and you'd like to design a system that uses WA/NT data, you're welcome to contribute, start a conversation on the github page. Alternatively, simply build a HVDC link across the Nullarbor and you can save yourself the ML headache.
Could this be done in the USA? Europe?
Yes! However, it will take quite a bit of effort. Each different market (there are 10+ in the USA alone) provides similar data but in different formats. If you're interested, get in contact.
How are Interconnectors treated - and why isn't Tasmania always 100% Greenness?
Interconnectors?The big transmission lines that link states together affect Greenness and price. Electricity imported from another state is assumed to have the same Greenness as the state it's coming from - it's split into "imported green energy" and "imported fossil energy". This is why Tasmania, with no significant fossil plants, isn't always at 100% Greenness - imported electricity from Victoria has lower Greenness. Similarly, Victoria is more Green when it is importing from Tasmania.
What is "Wholesale Price"?
Power companies and power plants buy and sell electricity to each other on a "national" spot market, operated by AEMO. Every 5 minutes there is a new market price for each region (state). Every-day consumers typically don't have direct access to this market: their power company buys on behalf of all its customers. Customers pay a flat fee to the power company so the minute-to-minute fluctations of the market don't affect them directly. However, larger commercial/industrial customers may be directly exposed to market fluctuations and there are also some (TODO) power companies that offer 'pass-through' pricing to households.
Note that the wholesale price of electricity only makes up part of your power bill: other "fixed costs" for infrastructure eg "poles and wires" also contribute.
When are forecasts less accurate?
- If the weather forecasts turn out to be wrong then wind and solar projections will also be wrong. If the weather forecast for some day changes as it gets closer, the model will adjust Greenness and Price predictions too.
- If there's an unexpected outage of a large power plant or interconnector then predictions will be wrong. This affects price predictions more than Greenness. Planned outages are taken into account.
- Times of high market volatility (eg 2022) result in lower-accuracy price forecasts
- This model is not specifically designed to predict the large price spikes that sporadically occur; this would not contribute to this project's goals.
How does this compare to other predictions?
Greenness: watttime.org uses different measure in place of Greenness: a marginal rate not an absolute one (see their methodology and API pages).
Price: AEMO provides "forecasts"?Predispatch arguably isn't intended as a forecast but is often used as such up to 48h in the future. Standing on these shoulders, the GreenForecast.au improves these predictions a little. Beyond 48 hours, grenforecast.au seems to be the only one.
If you would like to compare these predictions to your own forecaster or you know of any public alternatives, please get in touch! Mail info[AT]greenforecast.au or @GreenForecastAu on twitter.
How does performance compare across states?
NSW and VIC are more accurate, laregly due to their size (stability). SA is the hardest: it is a smaller market and has the highest proportion of unpredictable generation.
How has the 2022 energy crisis affected predictions?
In 2022, the market is (a) more volatile and (b) has higher average prices. In particular, the market 'shutdown' in June 2022. The model certainly didn't predict that, though Greenness predictions were still pretty good.
From a machine learning point of view, the model did a mediocre job of predicting post-crisis prices (about 3x worse) when it was trained on only pre-crisis data. However, now that post-crisis data is available to train on, the model has a similar accuracy (in percentage terms) to before.
How is the rapid growth of renewables affecting the model's accuracy?
Not much, because this growth is already reflected in the raw data coming from AEMO. Of course, Greenness factors are now much higher than they were!
Why split state-by-state? Don't local effects matter?
There is no great reason for this. The NEM is split up this way and so is the data.
Q&A From Tech People:
Is there an API available?
Please get in contact, see links in the footer. I'd love to help.
What kind of model is it?
For Greenness, a simple neural net (MLP) is trained for each timestep (ie 2 hours in the future, 4 hours in future, ...). The models predict the outputs of each fuel type, from which greenness is derived as a simple proportion.
For price, the models just predict the target value. The neural nets are ensembled with an XGBoost model for each timestep for a small (~5%) improvement in performance. This is all repeated for price and Greenness for each state, for a total of around a thousand sub-models.
What are the model inputs?
Roughly a thousand features are collected at inference. Data comes from AEMO, the BoM, a government public holidays database and several derived values.
What is the training dataset?
A custom-prepared dataset with around one million datapoints for each NEM region dating back to 2010. Some basic (hand-crafted) data augmentation bulks out scarce post-2022-energy-crisis data. For details, see TODO on github.
What types features are used?
Date features, current/forecast weather, current/forecast NEM data (generation & availability by fuel type, price, interconnectors), price lags and Greenness lags.
Each sub-model (eg Price 48h in advance) uses a hand-crafted subset of around 100 features - largely just excluding lags that are less relevant. For example, when forecasting 48h in advance, the weather forecast 5 days in the future is irrelevant.
How does this model compare to architecture X?
Experiments so far: Random Forests, ARIMA, DeepAR, TFT, NHiTS, NBEATS, LGBoost, Prophet, various variations on MLPs. Some experiments were extensive, others weren't. Simple MLPs and XGBoost have performed the best. [results TODO]. Suggestions, feedback and experimentation with alternatives welcome.
Why might this architecture be performing best?
This problem may differ from standard time-series forecasting problems (eg M5) because there are good covariant features and forecasts (eg weather, wind output, etc) available. With basic feature engineering (eg selected past and forecast lags) it perhaps looks more like a tabular data problem than standard time-series forecasting. If so, strong performance from simple neural networks and gradient boosting is not surprising. TODO
What other techniques were used to improve results?
In rough order of effectiveness:
- As always, getting more data (longer timespan and extra features) made the biggest difference. For example: Some solar data is not available before 2016, but a random forests regressor does a great job of predicting this data back to 2010 from other available data (weather, installed base). See RooftopBackcast.ipynb in the dataset generator.
- Basic data augmentation - interpolating between datapoints and duplicating points with some features scaled in a logical way (eg all price features increased by 5%)
- Training each sub-model on a preselected subset of features, eg the submodel predicting 48 hours ahead was not shown forecasts for 96 hours ahead. This minimises "uninformative features"
- Encoding date features as numeric rather than embeddings improved accuracy, again reducing uninformative features.
- Using a holdout test set to choose architecture and hyperparameters but making that holdout data available for final training. Held out data is the most recent available and therefore the most releveant to today's market behaviour. A validation set was still used to prevent overfitting.
- Various training tricks promoted by fast.ai
Can I try my own model with the dataset?
Yes! Currently you'll need to run the dataset generator yourself, the datasets themselves are not yet uploaded. Please share your results! Contact below.
Which metrics are used?
From most-convenient to most-comprehensive:
- MAE for price 84h in the future on VIC1 dataset (ie the feature named 'VIC1_Price_Tp84'). This is selected because price is more variable than Greenness, the time is in middle of the range and not an even multiple of 24h
- MAE for all the various fuel sources 84h ahead
- MAE for price for all timesteps from 2h to 168h ahead for VIC1 dataset
- MAE for price across all timesteps and all regions
- MAE for price and Greenness separately for all timesteps and all regions
There's no strong reason for choosing MAE over MSE - brief comparison experiments were inconclusive. Price forecasts vary across several orders of magnitude (even after clamping values to ±$1000/MWh) so MSE would only emphasise that further. This project prioritises the 'every-day' case over outlier events, so MAE wins from that point of view.
Why not use the same model/ensemble for each state?
Bespoke models for each state performed better than a single model trained on all states (perhaps because there is already a million datapoints per state to train on) or a single model that predicts all states at once (perhaps because this greatly increases the number of features)
Can I use the dataset for my own project?
Yes. Github TODO (See contact details below for now)
How long does training take?
Training a thousand sub-models takes about 3 days on a single consumer GPU. Inference takes around ten minutes, most of which is gathering data.
How is the model hosted?
Dataset generation and model training are offline. Inference is in AWS Lambda, kicking off every two hours. This website frontend is just a static page on S3. For details, see the deployment diagram on github.