Enhancing temporal variability of 5-minute satellite-
derived solar irradiance data
1Jing Huang, 2Richard Perez, 2James Schlemmer, 1Alex Kubiniec, 1Marc Perez, 1Akanksha Bhat and
1Clean Power Research, Napa, CA, USA
2Atmospheric Sciences Research Center, SUNY, Albany, NY, USA
Abstract—Satellite-derived solar irradiance data are known to
underestimate temporal variability compared to point
measurements because of their pixel-averaging nature. In this
study, we apply an algorithm imposing random noise to enhance
the temporal variability of 5-minute satellite-derived solar
irradiance data. We show that the resulting product, termed as
True Dynamics, has clear-sky exceedance events and the
frequency of large ramp events closer to observation. In addition,
the increase of temporal resolution of irradiance data significantly
reduces the underestimation error of power inverter clipping
under high DC:AC capacity ratios conditions.
Keywords—solar irradiance, SolarAnywhere, variability,
satellite, photovoltaics, inverter clipping
Spatial and temporal variability caused by events such as
cloud and aerosol passage is one of the inherent characteristics
of solar irradiance data. This has cascading effects on estimating
and forecasting solar power generation at various aggregation
levels. Ideally, the solar industry desires high-quality solar
resource data at fine spatial (e.g. sub-km) and temporal (e.g. sub-
minute) scales to capture variability patterns for accurate
bottom-up solar power modeling. Although this goal has been
gradually approached mainly by development in remote sensing
and image processing techniques, complementary empirical
approaches attempting to reconstruct spatiotemporal solar
variabilities are still useful.
Perez et al.  parameterized intra-hourly solar variabilities
using four key metrics distilled from measurement data at 24
sites across the United States, which includes the standard
deviation of the global irradiance clear sky index, and the mean
index change from one time interval to the next, as well as the
maximum and standard deviation of the latter. The clear-sky
index, or kt, which is defined as Global Horizontal Irradiance
(GHI) divided by GHI at a presumptive clear-sky condition. In
this study, we apply this approach to the latest SolarAnywhere®
(SA hereafter)  V3.5 data product, which will be operational
at 5-min interval covering the entire United States. We show that
this approach can artificially enhance the temporal variability
and create clear-sky exceedance events that match better with
observation at 10 ground stations in the eastern US. In addition,
we quantify the effects of both irradiance temporal resolution
and empirical variability enhancement on the estimation of solar
photovoltaic (PV) power as a function of PV systems’ DC:AC
A. Ground stations for validation
Figure 1 and Table 1 provide information on the reference
ground stations used in this study. The measurement data span
the entire year of 2020 with a temporal resolution of 1 minute,
which can then be down-sampled to a resolution of 5 minutes to
match the resolution of the SA data.
Figure 1. 10 reference ground stations in the eastern United States are
used in this study.
Table 1. Metadata of the 10 reference ground stations
B. SolarAnywhere® V3.5
SA provides bankable solar irradiance data to support the
growth of solar industry globally. The latest version V3.5 which
was introduced May 2021, uses 3-hourly aerosol optical depth
(AOD) data from MERRA-2 to account for aerosol variability
in modeling clear-sky irradiance. This represents a key
operational improvement compared to the previous version V3.4
that used only monthly climatological aerosol data. The
resulting difference in clear-sky Direct Normal Irradiance (DNI)
is significant particularly during onset and passage of intense
aerosol events such as wildfires as shown for the Boulder site
from 08/2020 to 10/2020. In addition, the temporal resolution
of SA V3.5 will increase from 30 minutes to 5 minutes
Thus, we evaluate the 5-min V3.5 product in this study. Because
of the improved clear-sky modeling in V3.5, the resulting kt has
a better representation of cloud opacity. Figure 3 shows a good
match between SolarAnywhere and ground measurements. The
deviations from the 1:1 line are genesymmetric, except when
GHI is high (i.e. > 900 W m-2). This is because the SA irradiance
model, like other semi-empirical satellite models, assume that
irradiance quantities are capped at their clear-sky values [4,5]
whilst the measurments do contain clear-sky exceedence events.
Figure 2. (top) Annual variations of daily mean clear-sky DNI for the
Boulder site in 2020. The smoother line is V3.4 and the more
variable line is V3.5; (bottom) The corresponding daily mean AOD
at 550nm retrieved from MERRA-2.
Figure 3. 2D histogram between 5-min SolarAnywhere® V3.5
Global Horizontal Irradiance (GHI) and 5-min pyranometer-
measured GHI for the aggregation of 10 stations.
C. Enhancing variability of global irradiance at 5-min
To reconcile the extent of temporal variability of the SA
V3.5 with measurements which includes clear-sky exceedance
events, we impose random noises onto SA V3.5 irradiance data
based on parameterizations in Perez et al. . These clear sky
exceedance events are modeled by randomly enhancing the
satellite-derived clearness index changes from one five-minute
interval to the next (Δkt) under conditions that exhibit both high
variabilities as determined from the satellite-derived Δkt and a
relatively high irradiance level, as determined by the value of
kt. Action thresholds are defined for both Δkt and kt, above
which high-frequency (HF) random enhancement is applied.
The amplitude of the enhancement is fitted to observations so
that amplified Δkt distributions approach observations.
In addition to the HF variability enhancement, low-
frequency (LF) background amplification is also conducted.
This step is designed to enhance the effectiveness of accurately
capturing losses in all inverter-limited (DC:AC) cases, not by
only accounting for short-term spikes but also accounting for
prolonged conditions when irradiance can remain higher than
clear sky (e.g., from stable cloud enhancement) and vice-versa
without affecting overall bias.
To distinguish from the raw 5-min SA data, here the
variability enhanced 5-min product is termed as True Dynamics
(TD). A preview of TD is provided in Figure 4. By visual
inspection this product adds clear-sky exceedance events whilst
enhancing its temporal variability noticeably. In particular, note
the sustained higher irradiance levels near 2pm at bottom
compared to the HF-only enhanced model at bottom middle.
Figure 4. An example day (Jun 29th, 2020) of 5-min GHI time series
at Boulder comparing GHI time series between (top) measurement,
(top middle) raw SA, (bottom middle) high-frequency only amplified
SA and (bottom) fully amplified SA with both high-frequency and
low-frequency components. The unit of GHI is W m-2.
II. THE EFFECTS OF VARIABILITY ENHANCEMENT
After the variability-enhanced time series data pass visual
inspection, they need to be examined in more details from the
perspectives of distribution and accuracy.
Figure 5 illustrates the effects of variability enhancement in
terms of distribution. It can be discerned that SA generates kt
more frequently in the range of (0.75, 1] whilst the variability
enhancement procedure is able to alleviate this over-population. In
addition, the variability-enhanced kt has occurrences in the
clear-sky exceedance bin (1, 1.25] that do not exist in the raw SA
Figure 5. Histogram box plots of 10 sites for (top) kt and (bottom) Δkt.
The bins for kt are (0, 0.25], (0.25, 0.5], (0.5, 0.75], (0.75, 1], (1, 1.25].
The bins for Δkt are (0, 0.1], (0.1, 0.2], (0.2, 0.3], (0.3, 0.4], (0.4, 0.5].
Note that the histogram of Δkt is plotted in log scale to highlight the
comparison of large ramp events.
An important characteristic of temporal variability is the
frequency of ramp events. It is clear from the Δkt plot in Figure 5
that whilst the raw SA kt has less frequent large ramp events than
measurements, the variability-enhancing algorithm is able to
increase the frequency of large ramp events to the observed
extent. This feature would be useful for scenarios where ramp
events play a key role such as hybrid PV-battery systems
providing pre-scheduled power generation or stability and
adequacy testing for ISO system planning.
B. Error metrics
It is promising that our artificial enhancement of variability
has created clear-sky exceedance events that do not exist in raw
satellite data and increased the frequency of large ramp events.
However, due to the random nature of modifying the satellite-
derived kt in the algorithm, the amplified kt becomes inherently
less accurate than the raw kt. To quantify the tradeoff between
variability and accuracy, we employ Taylor Diagram on the raw
kt and the modified kt signal.
Taylor Diagram was invented by Taylor (2001) originally
for geophysical studies. It displays three important and
connected metrics within just one graph. Assuming we have a
general reference field r, and a test field t, the three metrics are
connected by the following equation:
, are the standard deviation of the error, the
reference field, and the test field, respectively, and is the
correlation coefficient between the reference and the test field.
Figure 6. Taylor diagram illustrates three metrics (i.e. correlation
coefficients, centered RMSE and standard deviation) of the raw SA
kt (circles) and the amplified SA kt (triangles).
In our case, the reference field is kt derived from
measurements. Note that the measured kt is normalized to have
a unity standard deviation such that the star position always
represents observation. The test field is the raw SA kt and the
amplified kt. It is clear from Figure 6 that the visual effect of
manually enhancing the variability of kt is to move the position
of a site generally upward in the Taylor diagram. This move
corresponds to an increased and , and a decreased . Since
an increasing implies higher temporal variability and an
increasing and a decreasing imply reduced accuracy, it can
be drawn that for all 10 sites, this step has served its purpose of
design to enhance temporal variability but at a price of reduced
III. POWER INVERTER CLIPPING
The industry-standard data requirement for solar energy
resource assessment has traditionally been hourly resolution.
However, this has been shown to be insufficient to inform
critical financial decisions particularly under scenarios where
the capacity of AC inverters is smaller than that of DC power
generation . Indeed, the lack of both intra-hour variability and
clear sky exceedance events, implies an underestimation of
clipping losses: actual data spikes near and above clear sky are
curtailed while modeled data remaining below clear sky are not.
As such, we examine what the availability of 5-min satellite-
derived irradiance data (raw and variability enhanced) imply to
addressing this issue.
To quantify the effect of time resolution on power
estimation, we build a power model using PVLIB for a
hypothetical horizontal single-axis tracking PV system at all 10
locations. This is a prevailing choice for new installation of
utility-scale solar PV farms due to its relative low cost and
excellent mechanical durability. The maximum rotation angle of
the tracking system is specified at 70 degrees with backtracking
capability. The PVWatts model is used for DC and AC with
default parameter setting . To model the temperature derating
effect, we use temperature site measurement and a coefficient of
-0.37% oC-1 which is representative of common multi-crystalline
PV cells. The separation of GHI into direct and diffuse
components are performed using the DIRINT model . The
overestimation of power by the use of hourly irradiance data
during variable irradiance and inverter clipping period is
illustrated in Figure 7.
Figure 7. Comparison of hourly averaged and 5-minute (top) GHI
time series data; (bottom) AC power yield normalized by the capacity
of the inverter. A DC:AC ratio of 1.8 is used for irradiance-to-power
conversion. The red circle indicates the period when the sub-hourly
variability of AC power is not accounted for by using hourly
Using the proposed configuration for the power model, we
are able to estimate AC power yield at various time resolutions.
To overcome inaccuracies due to numerical issues we first
linearly interpolate and align all data sources to 1-min
resolution. And then we run the power model for various DC:AC
scenarios. The curtailed power is calculated as P(DC:AC=1.0) –
P(DC:AC), where P is the output of the PVLIB power model. It
should be noted that the bulk bias in irradiance plays an
increasingly important role in the estimation of the curtailed
power as DC:AC increases. However, site-specific debiasing is
beyond the scope of this study and its effectiveness is not always
warranted. As such, we do not perform any debiasing operations
and rather assess the performance of the raw 5-min SA data and
the variability-enhanced 5-min SA data (i.e. TD). We regard the
power estimation using the 1-min measurement data as truth and
also use its total production to normalize the error in power
Figure 8. Box plots of relative error of curtailed AC power estimation
using 1-hour observation, 5-min observation, 1-hour SA, 5-min SA,
Figure 8 illustrates the effects of data sources and time
resolutions on the accuracy of estimating curtailed power due to
the limited capacity of AC power inverter. Compared to using
hourly averaged irradiance data, the use of 5-minute data is able
to generally reduce the underestimation of power inverter
clipping significantly. For example, under DC:AC=1.8, the 10-
site average of the clipped power error decreases from -3% when
using hourly SA data to -1% when using 5-min SA data.
Furthermore, with our variability treatment, the 5-min TD data
present an error of only 0.3% for 10-site average and
DC:AC=1.8 with an associated standard deviation of around
1%. Although it is inevitable that the bias in SA data plays a role
in this comparison, in particular under high DC:AC scenarios, it
is clear that our new 5-min SA data and our variability
enhancement technology does help reduce the error in
estimating power inverter clipping and hence total power
1. We have succeeded in implementing a variability
enhancing algorithm on 5-minute satellite-derived solar
irradiance data. As shown quantitatively, the algorithm
has created clear-sky exceedance events more
commensurate with measurement and increased the
frequency of large ramp events. However, it also
inevitably reduces short-term accuracy as measured by
correlation and RMSE. The amplified irradiance data
are suitable for scenarios where variability
characteristics are important. For example, the extent
and frequency of power ramp events play a key role in
the battery life and stability of a hybrid PV-battery
2. As shown, the amplified irradiance data (i.e. TD) are
also suitable for long-term PV yield estimation when the
DC:AC ratio is high. As an indication, the error of
estimating power clipping loss is only 0.3% for a 10-site
average under DC:AC=1.8.
3. We note that no debiasing schemes have been applied
for individual sites. With efforts dedicated to
addressing this, the 5-minute SA data and variability
enhancements have the potential to further improve the
performance of power clipping loss and the total PV
yield estimation and other applications where the
information of intra-hourly variabilities are required.
 R. Perez, S. Kivalov, J. Schlemmer, K. Hemker, and T. Hoff,
“Parameterization of site-specific short-term irradiance variability,” Sol.
Energy, 85 (7), 1343-1353, 2011
 K.E. Taylor, (2001). "Summarizing multiple aspects of model
performance in a single diagram". J. Geophys. Res. 106, 7183–7192,
 K. Bradford, R. Walker, D. Moon and M. Ibanez, “A regression model
to correct for intra-hourly irradiance variability bias in solar energy
models”, 2020 47th IEEE Photovoltaic Specialists Conference (PVSC),
2020, pp. 2679-2682.
 Perez R., P. Ineichen, K. Moore, M. Kmiecik, C. Chain, R. George and
F. Vignola, (2002): A New Operational Satellite-to-Irradiance Model.
Solar Energy 73, 5, pp. 307-317.
 Perez R., P. Ineichen, M. Kmiecik, K. Moore, R. George and D. Renne,
(2004): Producing satellite-derived irradiances in complex arid terrain.
Solar Energy 77, 4, 363-370
 Dobos A.P., (2014) “PVWatts Version 5 Manual”, NREL.
 Perez, R., P. Ineichen, E. Maxwell, R. Seals and A. Zelenka, (1992).
“Dynamic Global-to-Direct Irradiance Conversion Models”. ASHRAE
Transactions-Research Series, pp. 354-369