Conference PaperPDF Available

Enhancing temporal variability of 5-minute satellite- derived solar irradiance data

  • Clean Power Research

Abstract and Figures

Satellite-derived solar irradiance data are known to underestimate temporal variability compared to point measurements because of their pixel-averaging nature. In this study, we apply an algorithm imposing random noise to enhance the temporal variability of 5-minute satellite-derived solar irradiance data. We show that the resulting product, termed as True Dynamics, has clear-sky exceedance events and the frequency of large ramp events closer to observation. In addition, the increase of temporal resolution of irradiance data significantly reduces the underestimation error of power inverter clipping under high DC:AC capacity ratios conditions.
Content may be subject to copyright.
Enhancing temporal variability of 5-minute satellite-
derived solar irradiance data
1Jing Huang, 2Richard Perez, 2James Schlemmer, 1Alex Kubiniec, 1Marc Perez, 1Akanksha Bhat and
1Patrick Keelin
1Clean Power Research, Napa, CA, USA
2Atmospheric Sciences Research Center, SUNY, Albany, NY, USA
AbstractSatellite-derived solar irradiance data are known to
underestimate temporal variability compared to point
measurements because of their pixel-averaging nature. In this
study, we apply an algorithm imposing random noise to enhance
the temporal variability of 5-minute satellite-derived solar
irradiance data. We show that the resulting product, termed as
True Dynamics, has clear-sky exceedance events and the
frequency of large ramp events closer to observation. In addition,
the increase of temporal resolution of irradiance data significantly
reduces the underestimation error of power inverter clipping
under high DC:AC capacity ratios conditions.
Keywordssolar irradiance, SolarAnywhere, variability,
satellite, photovoltaics, inverter clipping
Spatial and temporal variability caused by events such as
cloud and aerosol passage is one of the inherent characteristics
of solar irradiance data. This has cascading effects on estimating
and forecasting solar power generation at various aggregation
levels. Ideally, the solar industry desires high-quality solar
resource data at fine spatial (e.g. sub-km) and temporal (e.g. sub-
minute) scales to capture variability patterns for accurate
bottom-up solar power modeling. Although this goal has been
gradually approached mainly by development in remote sensing
and image processing techniques, complementary empirical
approaches attempting to reconstruct spatiotemporal solar
variabilities are still useful.
Perez et al. [1] parameterized intra-hourly solar variabilities
using four key metrics distilled from measurement data at 24
sites across the United States, which includes the standard
deviation of the global irradiance clear sky index, and the mean
index change from one time interval to the next, as well as the
maximum and standard deviation of the latter. The clear-sky
index, or kt, which is defined as Global Horizontal Irradiance
(GHI) divided by GHI at a presumptive clear-sky condition. In
this study, we apply this approach to the latest SolarAnywhere®
(SA hereafter) [6] V3.5 data product, which will be operational
at 5-min interval covering the entire United States. We show that
this approach can artificially enhance the temporal variability
and create clear-sky exceedance events that match better with
observation at 10 ground stations in the eastern US. In addition,
we quantify the effects of both irradiance temporal resolution
and empirical variability enhancement on the estimation of solar
photovoltaic (PV) power as a function of PV systems’ DC:AC
capacity ratios.
A. Ground stations for validation
Figure 1 and Table 1 provide information on the reference
ground stations used in this study. The measurement data span
the entire year of 2020 with a temporal resolution of 1 minute,
which can then be down-sampled to a resolution of 5 minutes to
match the resolution of the SA data.
Figure 1. 10 reference ground stations in the eastern United States are
used in this study.
Table 1. Metadata of the 10 reference ground stations
1661 m
Penn State
400 m
Fort Peck
686 m
Goodwin Creek
104 m
Sioux Falls
475 m
212 m
285 m
508 m
93 m
1546 m
B. SolarAnywhere® V3.5
SA provides bankable solar irradiance data to support the
growth of solar industry globally. The latest version V3.5 which
was introduced May 2021, uses 3-hourly aerosol optical depth
(AOD) data from MERRA-2 to account for aerosol variability
in modeling clear-sky irradiance. This represents a key
operational improvement compared to the previous version V3.4
that used only monthly climatological aerosol data. The
resulting difference in clear-sky Direct Normal Irradiance (DNI)
is significant particularly during onset and passage of intense
aerosol events such as wildfires as shown for the Boulder site
from 08/2020 to 10/2020. In addition, the temporal resolution
of SA V3.5 will increase from 30 minutes to 5 minutes
operationally soon.
Thus, we evaluate the 5-min V3.5 product in this study. Because
of the improved clear-sky modeling in V3.5, the resulting kt has
a better representation of cloud opacity. Figure 3 shows a good
match between SolarAnywhere and ground measurements. The
deviations from the 1:1 line are genesymmetric, except when
GHI is high (i.e. > 900 W m-2). This is because the SA irradiance
model, like other semi-empirical satellite models, assume that
irradiance quantities are capped at their clear-sky values [4,5]
whilst the measurments do contain clear-sky exceedence events.
Figure 2. (top) Annual variations of daily mean clear-sky DNI for the
Boulder site in 2020. The smoother line is V3.4 and the more
variable line is V3.5; (bottom) The corresponding daily mean AOD
at 550nm retrieved from MERRA-2.
Figure 3. 2D histogram between 5-min SolarAnywhere® V3.5
Global Horizontal Irradiance (GHI) and 5-min pyranometer-
measured GHI for the aggregation of 10 stations.
C. Enhancing variability of global irradiance at 5-min
To reconcile the extent of temporal variability of the SA
V3.5 with measurements which includes clear-sky exceedance
events, we impose random noises onto SA V3.5 irradiance data
based on parameterizations in Perez et al. [1]. These clear sky
exceedance events are modeled by randomly enhancing the
satellite-derived clearness index changes from one five-minute
interval to the next (Δkt) under conditions that exhibit both high
variabilities as determined from the satellite-derived Δkt and a
relatively high irradiance level, as determined by the value of
kt. Action thresholds are defined for both Δkt and kt, above
which high-frequency (HF) random enhancement is applied.
The amplitude of the enhancement is fitted to observations so
that amplified Δkt distributions approach observations.
In addition to the HF variability enhancement, low-
frequency (LF) background amplification is also conducted.
This step is designed to enhance the effectiveness of accurately
capturing losses in all inverter-limited (DC:AC) cases, not by
only accounting for short-term spikes but also accounting for
prolonged conditions when irradiance can remain higher than
clear sky (e.g., from stable cloud enhancement) and vice-versa
without affecting overall bias.
To distinguish from the raw 5-min SA data, here the
variability enhanced 5-min product is termed as True Dynamics
(TD). A preview of TD is provided in Figure 4. By visual
inspection this product adds clear-sky exceedance events whilst
enhancing its temporal variability noticeably. In particular, note
the sustained higher irradiance levels near 2pm at bottom
compared to the HF-only enhanced model at bottom middle.
Figure 4. An example day (Jun 29th, 2020) of 5-min GHI time series
at Boulder comparing GHI time series between (top) measurement,
(top middle) raw SA, (bottom middle) high-frequency only amplified
SA and (bottom) fully amplified SA with both high-frequency and
low-frequency components. The unit of GHI is W m-2.
After the variability-enhanced time series data pass visual
inspection, they need to be examined in more details from the
perspectives of distribution and accuracy.
A. Distributions
Figure 5 illustrates the effects of variability enhancement in
terms of distribution. It can be discerned that SA generates kt
more frequently in the range of (0.75, 1] whilst the variability
enhancement procedure is able to alleviate this over-population. In
addition, the variability-enhanced kt has occurrences in the
clear-sky exceedance bin (1, 1.25] that do not exist in the raw SA
Figure 5. Histogram box plots of 10 sites for (top) kt and (bottom) Δkt.
The bins for kt are (0, 0.25], (0.25, 0.5], (0.5, 0.75], (0.75, 1], (1, 1.25].
The bins for Δkt are (0, 0.1], (0.1, 0.2], (0.2, 0.3], (0.3, 0.4], (0.4, 0.5].
Note that the histogram of Δkt is plotted in log scale to highlight the
comparison of large ramp events.
An important characteristic of temporal variability is the
frequency of ramp events. It is clear from the Δkt plot in Figure 5
that whilst the raw SA kt has less frequent large ramp events than
measurements, the variability-enhancing algorithm is able to
increase the frequency of large ramp events to the observed
extent. This feature would be useful for scenarios where ramp
events play a key role such as hybrid PV-battery systems
providing pre-scheduled power generation or stability and
adequacy testing for ISO system planning.
B. Error metrics
It is promising that our artificial enhancement of variability
has created clear-sky exceedance events that do not exist in raw
satellite data and increased the frequency of large ramp events.
However, due to the random nature of modifying the satellite-
derived kt in the algorithm, the amplified kt becomes inherently
less accurate than the raw kt. To quantify the tradeoff between
variability and accuracy, we employ Taylor Diagram on the raw
kt and the modified kt signal.
Taylor Diagram was invented by Taylor (2001) originally
for geophysical studies. It displays three important and
connected metrics within just one graph. Assuming we have a
general reference field r, and a test field t, the three metrics are
connected by the following equation:
 ()
where ,
, are the standard deviation of the error, the
reference field, and the test field, respectively, and is the
correlation coefficient between the reference and the test field.
Figure 6. Taylor diagram illustrates three metrics (i.e. correlation
coefficients, centered RMSE and standard deviation) of the raw SA
kt (circles) and the amplified SA kt (triangles).
In our case, the reference field is kt derived from
measurements. Note that the measured kt is normalized to have
a unity standard deviation such that the star position always
represents observation. The test field is the raw SA kt and the
amplified kt. It is clear from Figure 6 that the visual effect of
manually enhancing the variability of kt is to move the position
of a site generally upward in the Taylor diagram. This move
corresponds to an increased and , and a decreased . Since
an increasing implies higher temporal variability and an
increasing  and a decreasing imply reduced accuracy, it can
be drawn that for all 10 sites, this step has served its purpose of
design to enhance temporal variability but at a price of reduced
The industry-standard data requirement for solar energy
resource assessment has traditionally been hourly resolution.
However, this has been shown to be insufficient to inform
critical financial decisions particularly under scenarios where
the capacity of AC inverters is smaller than that of DC power
generation [3]. Indeed, the lack of both intra-hour variability and
clear sky exceedance events, implies an underestimation of
clipping losses: actual data spikes near and above clear sky are
curtailed while modeled data remaining below clear sky are not.
As such, we examine what the availability of 5-min satellite-
derived irradiance data (raw and variability enhanced) imply to
addressing this issue.
To quantify the effect of time resolution on power
estimation, we build a power model using PVLIB for a
hypothetical horizontal single-axis tracking PV system at all 10
locations. This is a prevailing choice for new installation of
utility-scale solar PV farms due to its relative low cost and
excellent mechanical durability. The maximum rotation angle of
the tracking system is specified at 70 degrees with backtracking
capability. The PVWatts model is used for DC and AC with
default parameter setting [6]. To model the temperature derating
effect, we use temperature site measurement and a coefficient of
-0.37% oC-1 which is representative of common multi-crystalline
PV cells. The separation of GHI into direct and diffuse
components are performed using the DIRINT model [7]. The
overestimation of power by the use of hourly irradiance data
during variable irradiance and inverter clipping period is
illustrated in Figure 7.
Figure 7. Comparison of hourly averaged and 5-minute (top) GHI
time series data; (bottom) AC power yield normalized by the capacity
of the inverter. A DC:AC ratio of 1.8 is used for irradiance-to-power
conversion. The red circle indicates the period when the sub-hourly
variability of AC power is not accounted for by using hourly
irradiance data.
Using the proposed configuration for the power model, we
are able to estimate AC power yield at various time resolutions.
To overcome inaccuracies due to numerical issues we first
linearly interpolate and align all data sources to 1-min
resolution. And then we run the power model for various DC:AC
scenarios. The curtailed power is calculated as P(DC:AC=1.0)
P(DC:AC), where P is the output of the PVLIB power model. It
should be noted that the bulk bias in irradiance plays an
increasingly important role in the estimation of the curtailed
power as DC:AC increases. However, site-specific debiasing is
beyond the scope of this study and its effectiveness is not always
warranted. As such, we do not perform any debiasing operations
and rather assess the performance of the raw 5-min SA data and
the variability-enhanced 5-min SA data (i.e. TD). We regard the
power estimation using the 1-min measurement data as truth and
also use its total production to normalize the error in power
curtailment estimation.
Figure 8. Box plots of relative error of curtailed AC power estimation
using 1-hour observation, 5-min observation, 1-hour SA, 5-min SA,
5-min TD.
Figure 8 illustrates the effects of data sources and time
resolutions on the accuracy of estimating curtailed power due to
the limited capacity of AC power inverter. Compared to using
hourly averaged irradiance data, the use of 5-minute data is able
to generally reduce the underestimation of power inverter
clipping significantly. For example, under DC:AC=1.8, the 10-
site average of the clipped power error decreases from -3% when
using hourly SA data to -1% when using 5-min SA data.
Furthermore, with our variability treatment, the 5-min TD data
present an error of only 0.3% for 10-site average and
DC:AC=1.8 with an associated standard deviation of around
1%. Although it is inevitable that the bias in SA data plays a role
in this comparison, in particular under high DC:AC scenarios, it
is clear that our new 5-min SA data and our variability
enhancement technology does help reduce the error in
estimating power inverter clipping and hence total power
1. We have succeeded in implementing a variability
enhancing algorithm on 5-minute satellite-derived solar
irradiance data. As shown quantitatively, the algorithm
has created clear-sky exceedance events more
commensurate with measurement and increased the
frequency of large ramp events. However, it also
inevitably reduces short-term accuracy as measured by
correlation and RMSE. The amplified irradiance data
are suitable for scenarios where variability
characteristics are important. For example, the extent
and frequency of power ramp events play a key role in
the battery life and stability of a hybrid PV-battery
2. As shown, the amplified irradiance data (i.e. TD) are
also suitable for long-term PV yield estimation when the
DC:AC ratio is high. As an indication, the error of
estimating power clipping loss is only 0.3% for a 10-site
average under DC:AC=1.8.
3. We note that no debiasing schemes have been applied
for individual sites. With efforts dedicated to
addressing this, the 5-minute SA data and variability
enhancements have the potential to further improve the
performance of power clipping loss and the total PV
yield estimation and other applications where the
information of intra-hourly variabilities are required.
[1] R. Perez, S. Kivalov, J. Schlemmer, K. Hemker, and T. Hoff,
Parameterization of site-specific short-term irradiance variability,” Sol.
Energy, 85 (7), 1343-1353, 2011
[2] K.E. Taylor, (2001). "Summarizing multiple aspects of model
performance in a single diagram". J. Geophys. Res. 106, 71837192,
[3] K. Bradford, R. Walker, D. Moon and M. Ibanez, “A regression model
to correct for intra-hourly irradiance variability bias in solar energy
models”, 2020 47th IEEE Photovoltaic Specialists Conference (PVSC),
2020, pp. 2679-2682.
[4] Perez R., P. Ineichen, K. Moore, M. Kmiecik, C. Chain, R. George and
F. Vignola, (2002): A New Operational Satellite-to-Irradiance Model.
Solar Energy 73, 5, pp. 307-317.
[5] Perez R., P. Ineichen, M. Kmiecik, K. Moore, R. George and D. Renne,
(2004): Producing satellite-derived irradiances in complex arid terrain.
Solar Energy 77, 4, 363-370
[6] Dobos A.P., (2014) “PVWatts Version 5 Manual, NREL.
[7] Perez, R., P. Ineichen, E. Maxwell, R. Seals and A. Zelenka, (1992).
“Dynamic Global-to-Direct Irradiance Conversion Models”. ASHRAE
Transactions-Research Series, pp. 354-369
... We use PVLIB to model a hypothetical horizontal single-axis tracking PV system, and then calculate the power loss due to inverter clipping under various DC:AC ratios. We follow the same technical procedures as described in our companion paper [7]. Figure 8 demonstrates the superiority of the downscaled 1-min time series over the hourly averaged observation data for those scenarios with high DC:AC ratios. ...
Conference Paper
Full-text available
Small-scale variabilities of solar irradiance are important for many applications. Downscaling approaches need to be developed where only the averaged state of solar irradiance is known. In this study, we investigate the use of copula for temporally downscaling GHI clear-sky indices. With the correlation structure and distribution information derived from measurements at 10 stations across the United States, the copula approach is capable of downscaling clear-sky indices from hourly averages to any arbitrary fine scale whilst preserving its original power spectra.
Full-text available
This paper presents two models for converting hourly global into hourly direct beam irradiance based on a new parameterization of isolation conditions. The models are derived statistically from a large multiclimate experimental data base. The two models differ technically but should be functionally equivalent: The first is a correction to the physically based DISC model (Maxwell 1987); the second consists of a set of simple linear relationships. Validation results are presented in reference to three existing models. The validation benchmarks consist of root mean square and mean bias errors as well as the ability of the models to recreate the skewness and kurtosis of actual direct distributions.
We present a new simple model capable of exploiting geostationary satellite visible images for the production of site/time-specific global and direct irradiances The new model features new clear sky global and direct irradiance functions, a new cloud-index-to-irradiance index function, and a new global-to-direct-irradiance conversion model. The model can also exploit operationally available snow cover resource data, while deriving local ground specular reflectance characteristics from the stream of incoming satellite data. Validation against 10 US locations representing a wide range of climatic environments indicates that model performance is systematically improved, compared to current visible-channel-based modeling practice.
This article presents a set of empirical models capable of extracting metrics quantifying the short-term variability of the solar resource based upon site/time specific satellite-derived hourly irradiance data. The model is derived from over 92,000 experimental hourly data points at twenty-four sites in the United States. The model returns four metrics characterizing intra-hourly variability, including the standard deviation of the global irradiance clear sky index, and the mean index change from one time interval to the next, as well as the maximum and standard deviation of the latter. The variability time scales addressed in this paper are 20s, one-minute, 5min and 15min. The trends underlying the models are robust and show little site dependency.
This paper describes a methodology to correct satellite-derived irradiances over complex terrain for models that use the visible satellite channel as main input for cloud index determination. Complex terrain is characterized by high reflectance surface and or the juxtaposition of high and low reflectance surfaces (e.g., desert plains and forested ridges). The correction consists of (1) climate dependent post-model clear sky calibration and (2) singularity identification and removal.
Payback is often used as a measure of profitability by prospective PV owners. Contrasting this measure with another financial gauge––life-cycle cash flow––the paper discusses why payback may not be the most appropriate measure for residential PV applications and why it may hide sound financial opportunities for those deciding to invest in a PV system.In parallel, the paper addresses another aspect of economic feasibility: the value of energy produced. For residential applications, this value is currently set at residential net-metered retail rates. We present preliminary evidence that a higher value, reflective of the local effective capacity of PV ought to be claimed for residential PV applications.
A diagram has been devised that can provide a concise statistical summary of how well patterns match each other in terms of their correlation, their root-mean-square difference, and the ratio of their variances. Although the form of this diagram is general, it is especially useful in evaluating complex models, such as those used to study geophysical phenomena. Examples are given showing that the diagram can be used to summarize the relative merits of a collection of different models or to track changes in performance of a model as it is modified. Methods are suggested for indicating on these diagrams the statistical significance of apparent differences and the degree to which observational uncertainty and unforced internal variability limit the expected agreement between model-simulated and observed behaviors. The geometric relationship between the statistics plotted on the diagram also provides some guidance for devising skill scores that appropriately weight among the various measures of pattern correspondence.
PVWatts Version 5 Manual
  • A P Dobos
Dobos A.P., (2014) "PVWatts Version 5 Manual", NREL.