Zhenge Zhao’s research while affiliated with The University of Arizona and other places

What is this page?


This page lists works of an author who doesn't have a ResearchGate profile or hasn't added the works to their profile yet. It is automatically generated from public (personal) data to further our legitimate goal of comprehensive and accurate scientific recordkeeping. If you are this author and want this page removed, please let us know.

Publications (8)


Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models
  • Article

September 2021

·

14 Reads

·

45 Citations

IEEE Transactions on Visualization and Computer Graphics

Zhenge Zhao

·

Panpan Xu

·

Carlos Scheidegger

·

Liu Ren

The interpretation of deep neural networks (DNNs) has become a key topic as more and more people apply them to solve various problems and making critical decisions. Concept-based explanations have recently become a popular approach for post-hoc interpretation of DNNs. However, identifying human-understandable visual concepts that affect model decisions is a challenging task that is not easily addressed with automatic approaches. We present a novel human-in-the-loop approach to generate user-defined concepts for model interpretation and diagnostics. Central to our proposal is the use of active learning, where human knowledge and feedback are combined to train a concept extractor with very little human labeling effort. We integrate this process into an interactive system, ConceptExtract. Through two case studies, we show how our approach helps analyze model behavior and extract human-friendly concepts for different machine learning tasks and datasets and how to use these concepts to understand the predictions, compare model performance and make suggestions for model refinement. Quantitative experiments show that our active learning approach can accurately extract meaningful visual concepts. More importantly, by identifying visual concepts that negatively affect model performance, we develop the corresponding data augmentation strategy that consistently improves model performance.


Figure 2. The preliminary system utilizes a 2D heatmap (B) to visualize a time series across different floors (A). The variation of the color indicates the change of one physical attribute, which also reflects the vibration condition of the building. A linear interpolation is applied to the values between floors to facilitate comparisons across floors and different timestamps. The color encoding simplifies the recognition of three basic vibration modes in (C), nevertheless, this method lacks the ability of summarizing the simulation behavior as well as making comparisons across different simulations. See Section 4 for the technique we propose to solve these problems.
Figure 5. Our topic representation also helps connect to the concept of vibrational modes. Buildings vibrate mainly in the fundamental natural frequency (first mode) or as damage happens the floors may vibrate out of alignment (second, third modes). The topic representation helps to see if the floors are vibrating at the same frequencies, which can be verified if the signals are aligned by looking at the signal views.
Figure 6. The prototype system consists of four views. The matrix diagram (top left) is used for navigation and summarizes the overall behaviors across all the earthquakes. To understand the frequency distribution of each topic, the analysts can refer to the frequency spectrums (bottom left) for details. The opacity of the color indicates the relative magnitude of a frequency for a specific floor. The core of the system is the topic representation of each earthquake simulation (middle). It includes a content-based search module to help quickly identify similar partial time series across different simulations. The last part (right) is a details view that supports further exploration of simulation time series and helps civil engineers interpret the responses of buildings from another aspect.
Figure 7. The content-based search illustrates how topic modeling helps to identify regions that are locally similar and dissimilar. (A) and (C) show two different brushed regions in the top simulation and three of the most similar results aligned. For (A), a user can quickly see all three are similar hits and then validate this comparison in (B). For (C), the topmost hit ends up having a different behavior prior to the selection.
Figure 8. Analysts can select a portion of the ground acceleration (B) and drill down into a specific earthquake simulation (D), to visualize the response of a single physical variable plotted over time (x coordinate) and building floor (y coordinate). (C) is an area plot visualizing the selected portion of the ground acceleration. By utilizing STFT-LDA, analysts can quickly navigate and zoom in to the similar partial simulations. In (A) an analyst can quickly switch from EQ35 to EQ10 and zoom into time range 14.75s to 20.50s

+1

STFT-LDA: An Algorithm to Facilitate the Visual Analysis of Building Seismic Responses
  • Preprint
  • File available

September 2021

·

118 Reads

Zhenge Zhao

·

Danilo Motta

·

Matthew Berger

·

[...]

·

Carlos Scheidegger

Civil engineers use numerical simulations of a building's responses to seismic forces to understand the nature of building failures, the limitations of building codes, and how to determine the latter to prevent the former. Such simulations generate large ensembles of multivariate, multiattribute time series. Comprehensive understanding of this data requires techniques that support the multivariate nature of the time series and can compare behaviors that are both periodic and non-periodic across multiple time scales and multiple time series themselves. In this paper, we present a novel technique to extract such patterns from time series generated from simulations of seismic responses. The core of our approach is the use of topic modeling, where topics correspond to interpretable and discriminative features of the earthquakes. We transform the raw time series data into a time series of topics, and use this visual summary to compare temporal patterns in earthquakes, query earthquakes via the topics across arbitrary time scales, and enable details on demand by linking the topic visualization with the original earthquake data. We show, through a surrogate task and an expert study, that this technique allows analysts to more easily identify recurring patterns in such time series. By integrating this technique in a prototype system, we show how it enables novel forms of visual interaction.

Download

STFT-LDA: An algorithm to facilitate the visual analysis of building seismic responses

August 2021

·

19 Reads

·

3 Citations

Information Visualization

Civil engineers use numerical simulations of a building’s responses to seismic forces to understand the nature of building failures, the limitations of building codes, and how to determine the latter to prevent the former. Such simulations generate large ensembles of multivariate, multiattribute time series. Comprehensive understanding of this data requires techniques that support the multivariate nature of the time series and can compare behaviors that are both periodic and non-periodic across multiple time scales and multiple time series themselves. In this paper, we present a novel technique to extract such patterns from time series generated from simulations of seismic responses. The core of our approach is the use of topic modeling, where topics correspond to interpretable and discriminative features of the earthquakes. We transform the raw time series data into a time series of topics, and use this visual summary to compare temporal patterns in earthquakes, query earthquakes via the topics across arbitrary time scales, and enable details on demand by linking the topic visualization with the original earthquake data. We show, through a surrogate task and an expert study, that this technique allows analysts to more easily identify recurring patterns in such time series. By integrating this technique in a prototype system, we show how it enables novel forms of visual interaction.


Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models

August 2021

·

23 Reads

The interpretation of deep neural networks (DNNs) has become a key topic as more and more people apply them to solve various problems and making critical decisions. Concept-based explanations have recently become a popular approach for post-hoc interpretation of DNNs. However, identifying human-understandable visual concepts that affect model decisions is a challenging task that is not easily addressed with automatic approaches. We present a novel human-in-the-loop approach to generate user-defined concepts for model interpretation and diagnostics. Central to our proposal is the use of active learning, where human knowledge and feedback are combined to train a concept extractor with very little human labeling effort. We integrate this process into an interactive system, ConceptExtract. Through two case studies, we show how our approach helps analyze model behavior and extract human-friendly concepts for different machine learning tasks and datasets and how to use these concepts to understand the predictions, compare model performance and make suggestions for model refinement. Quantitative experiments show that our active learning approach can accurately extract meaningful visual concepts. More importantly, by identifying visual concepts that negatively affect model performance, we develop the corresponding data augmentation strategy that consistently improves model performance.


The ANTARES Astronomical Time-domain Event Broker

February 2021

·

30 Reads

·

72 Citations

The Astronomical Journal

We describe the Arizona-NOIRLab Temporal Analysis and Response to Events System (ANTARES), a software instrument designed to process large-scale streams of astronomical time-domain alerts. With the advent of large-format CCDs on wide-field imaging telescopes, time-domain surveys now routinely discover tens of thousands of new events each night, more than can be evaluated by astronomers alone. The ANTARES event broker will process alerts, annotating them with catalog associations and filtering them to distinguish customizable subsets of events. We describe the data model of the system, the overall architecture, annotation, implementation of filters, system outputs, provenance tracking, system performance, and the user interface.


Figure 4. Logical structure of the Alert Pipeline Worker. Input Alert streams are consumed at the top of the diagram and output streams are produced at the bottom.
Figure 5. Alert throughput over time as ANTARES processes 1.23 million ZTF alerts in approximately 11 hours. The data set was created by merging eight consecutive nights of ZTF data into a singe input stream for ANTARES.
Figure 6. Data from several days of operation showing the time that an alert spends in the ANTARES pipeline. Blue dots represent median values while red crosses show the maximum time spent in the pipeline for each of the operating windows plotted.
Figure 8. Viewing the details of a locus on the ANTARES Portal. Users can see a plot of the light curve of the object, a table of observed values, links to external catalogs, the positions of detections relative to the locus center, and a thumbnail view of the region powered by Aladdin Lite (Bonnarel et al. 2000; Boch & Fernique 2014).
ANTARES Provenance Log Content
The ANTARES Astronomical Time-Domain Event Broker

November 2020

·

170 Reads

We describe the Arizona-NOIRLab Temporal Analysis and Response to Events System (ANTARES), a software instrument designed to process large-scale streams of astronomical time-domain alerts. With the advent of large-format CCDs on wide-field imaging telescopes, time-domain surveys now routinely discover tens of thousands of new events each night, more than can be evaluated by astronomers alone. The ANTARES event broker will process alerts, annotating them with catalog associations and filtering them to distinguish customizable subsets of events. We describe the data model of the system, the overall architecture, annotation, implementation of filters, system outputs, provenance tracking, system performance, and the user interface.



Machine-learning-based Brokers for Real-time Classification of the LSST Alert Stream

January 2018

·

92 Reads

·

138 Citations

The Astrophysical Journal Supplement Series

The unprecedented volume and rate of transient events that will be discovered by the Large Synoptic Survey Telescope (LSST) demands that the astronomical community update its followup paradigm. Alert-brokers -- automated software system to sift through, characterize, annotate and prioritize events for followup -- will be critical tools for managing alert streams in the LSST era. The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is one such broker. In this work, we develop a machine learning pipeline to characterize and classify variable and transient sources only using the available multiband optical photometry. We describe three illustrative stages of the pipeline, serving the three goals of early, intermediate and retrospective classification of alerts. The first takes the form of variable vs transient categorization, the second, a multi-class typing of the combined variable and transient dataset, and the third, a purity-driven subtyping of a transient class. While several similar algorithms have proven themselves in simulations, we validate their performance on real observations for the first time. We quantitatively evaluate our pipeline on sparse, unevenly sampled, heteroskedastic data from various existing observational campaigns, and demonstrate very competitive classification performance. We describe our progress towards adapting the pipeline developed in this work into a real-time broker working on live alert streams from time-domain surveys.

Citations (5)


... Early research highlighted the need for example-level [1] and slice-level analysis [29,36,53]. Other tools employed interpretability methods to explain model predictions [9,14,22,45] and provided analysis at a conceptual-level or based on user-defined dimensions [27,28,50,56]. In addition, many tools have been designed to support model comparison [8,21,24,40,42,49]. ...

Reference:

LLM Comparator: Interactive Analysis of Side-by-Side Evaluation of Large Language Models
Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models
  • Citing Article
  • September 2021

IEEE Transactions on Visualization and Computer Graphics

... The strength of walls in the SW buildings is assumed to be constant along the height of the building while other systems (SMF, SCBF, and BRBF) have a varying SFRS strength as required to meet the code strength and deformation requirements along the height of buildings. The effects of wall strength, building height, and modal contributions on the seismic response of SW systems are studied in Kuzucu et al. (2018), Zhao et al. (2021), and Kuzucu et al. (2022). The plan and elevation views of SCBF, BRBF and SMF are illustrated in Figs. 2 and 5. ...

STFT-LDA: An algorithm to facilitate the visual analysis of building seismic responses
  • Citing Article
  • August 2021

Information Visualization

... These brokers ingest the alert streams and provide processed science products and services that enable the community to pursue their science goals without overloading the observatories' infrastructure. Current full-stream community brokers for ZTF and future LSST include ALeRCE (Förster et al. 2021), AMPEL (Nordin et al. 2019), ANTARES (Matheson et al. 2021), Babamul, Fink (Möller et al. 2020), Lasair (Smith et al. 2019), and Pitt-Google 1 . One of the persistent challenges is the identification and removal of contaminants (often referred to as "bogus" detections) from the observations and the transient event alert stream. ...

The ANTARES Astronomical Time-domain Event Broker
  • Citing Article
  • February 2021

The Astronomical Journal

... This motivates us to look at the clusters themselves. PCA, T-SNE, UMAP, or other more common statistical methods could be applied, however the interactivity of our approach fits well with "Visualizing Neural Networks with the Grand Tour" [7], and our project to extend it, started as a class project [8]. We have extended this into a tool we are calling n-dimensional scatter plot (NDSP), which is very helpful in making sense of and editing classifications in pixel distributions. ...

Visualizing Neural Networks with the Grand Tour
  • Citing Article
  • March 2020

Distill

... In Section 5, we summarize multiclass and binary classifier performance with and without redshift information, emphasizing accuracy as a function of classification confidence. We also consider performance on partial light curves for real-time classification through the ANTARES Broker (Narayan et al. 2018;Matheson et al. 2021). In Section 6, we compare Superphot+ʼs performance without redshift information to that of the ALeRCE (Sánchez-Sáez et al. 2021) light-curve classifier, one of the only comparable redshift-independent classifiers currently available in the literature. ...

Machine-learning-based Brokers for Real-time Classification of the LSST Alert Stream
  • Citing Article
  • January 2018

The Astrophysical Journal Supplement Series