Basic Analysis

The documentation below describes how to use the code base to analyze output from the U-net model on Animus. This guide assumes you have read through the Installation, Data, and Running the model pages and have model output available for analysis on Animus.

Table of Contents

Before you start
Exploring a dataset
Comparing inputs
Plotting results

back to top

Before you start

A good thing to do before starting is to check the version of the unox package you are using and the current working directory. The first time you run a cell in a notebook in VSCodium, it will ask you to select an environment to use. Be sure to select uplt, or whatever you named the environment during Installation.

import unox
print(unox.__version__)

0.1.1

If this command outputs an error, check where you are trying to run this notebook. By following the Installation, Data, and Running the model pages, this notebook will only run correctly on Animus.

import os
os.getcwd()

'/home/docs/checkouts/readthedocs.org/user_builds/unox/checkouts/stable/docs'

The current working directory for this example guide is within the docs/ directory. For this reason, most of the file paths below are prepended by ../ to indicate that you must first go up one directory level, then search for the file.

When doing analysis, I generally use a Jupyter Notebook. The notebook named plot_tests.ipynb is deliberately left blank as a place to test things out. I encourage you to follow along in that notebook as you go through this guide, noting any differences you need to keep track of based on the working directory and trying out different combinations that aren’t explicitly explored here.

back to top

Exploring a dataset

The code base contains a number of methods of exploring a dataset that is in a netCDF file. A number of sample datasets are listed below. Pick one to explore with this section.

# ERA5 U-wind component for 2019
sample_dataset = '../datafiles/sample_data/2019u10.nc'
# `t106` NOx emissions over North America for 2019
# sample_dataset = '../datafiles/sample_data/nox_2019_t106_US.nc'
# TROPESS reanalysis NOx for 2021
# sample_dataset = '../datafiles/sample_data/TROPESS_reanalysis_mon_emi_nox_anth_2021.nc'
# An example input netCDF
# sample_dataset = '../inputfiles/no2_2019_JFM/no2_2019_JFM.nc'
# An example output netCDF
# sample_dataset = '../HPC_runs/no2_example_run/predictions.nc'

back to top

Inspecting with map plots

While viewing the interactive text-based output above can give you a quick overview of what is in a dataset, it is often helpful to see plots of the data. NASA provides an application called Panoply which is a powerful tool for visualizing netCDF files containing geospatial data. If you are familiar with Panoply, please feel free to use it. Otherwise, the code base of unox includes several plotting functions which can quickly give an idea of what the dataset contains.

The cell below shows the use of a function to simply plot a box on a world map showing the maximum and minimum extents of the dataset in latitude and longitude.

from unox.plotting import plot_extent 

extent_plot = plot_extent(sample_dataset)

Matplotlib is building the font cache; this may take a moment.

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/proplot/__init__.py:6: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources as pkg

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/proplot/__init__.py:71: ProplotWarning: Rebuilding font cache. This usually happens after installing or updating proplot.
  register_fonts(default=True)

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/110m_physical/ne_110m_coastline.zip
  warnings.warn(f'Downloading: {url}', DownloadWarning)

_images/985e4b4a955acae62ec1747f5bc6fde31c912024e56738faa0c895575c1702e9.png

The next cell shows a function which plots a checkered grid on a map to show the rough resolution of the dataset. You can pass an optional keyword argument padding to change how much space is added around the extent of the dataset.

Note that this function is not currently optimized for datasets that cover the entire globe.

from unox.plotting import plot_lats_lons 

lats_lons_plot = plot_lats_lons(sample_dataset)

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_physical/ne_50m_coastline.zip
  warnings.warn(f'Downloading: {url}', DownloadWarning)

_images/a5ad26a3025aa5b367b3e0cdd057edd97b44be080da391c80af2b1e83956b7f6.png

The next few cells show examples of plotting the contents of particular variables from a dataset on a map. The specific variables (vars), which can be a list, will need to be selected such that they are within the given dataset. By default, the variables are averaged over the full time series.

from unox.plotting import plot_var_maps

var_plots = plot_var_maps(
    '../datafiles/sample_data/2019u10.nc',
    vars = 'u10',
)

/home/docs/checkouts/readthedocs.org/user_builds/unox/checkouts/stable/src/unox/HPC/data0/dataset.py:474: UserWarning: (is_ensemble) `dataset` must be a prediction set to check for ensemble members.
  warnings.warn(f"(is_ensemble) `dataset` must be a prediction set to check for ensemble members.")
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "

_images/f4663c147d1ee3a5c0fed5bfc5c326d1537094b4d03774d29f1eddb191647770.png

This function also has an optional argument called interval which, if given, will plot the temporal average across the specified time interval to average over starting at the start_date date. If start_date is not given, it will default to the first datetime in the dataset. The interval is specified by a string in the format XT where X is an integer and T is the time unit, either D for days or M for months.

from unox.plotting import plot_var_maps

var_plots = plot_var_maps(
    '../inputfiles/no2_2019_JFM/no2_2019_JFM.nc',
    vars = ['no2', 'no2_tm1', 'no2_s2', 'no2_s2_tm1'],
    start_date='2019-01-01',
    interval='2M',
)

/home/docs/checkouts/readthedocs.org/user_builds/unox/checkouts/stable/src/unox/HPC/data0/dataset.py:474: UserWarning: (is_ensemble) `dataset` must be a prediction set to check for ensemble members.
  warnings.warn(f"(is_ensemble) `dataset` must be a prediction set to check for ensemble members.")
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "

_images/fa816e883a7b00c305df1496cdbbef28e385caacce0662011a058553b42a1083.png

back to top

Comparing inputs

If you are generating new input files and would like to compare them to previous iterations, the plotting.compare_input_vars() function might be useful. In this function, you can specify two arrays to compare to each other by specifying:

input_set
- The name of the input file to use.
year
- The year for which to make the comparison.
var
- The variable for which to make the comparison.

The function will let you know how many values differ between the two specified arrays and, if there are differences, it will make plots to give an idea of what the differences are.

This was designed to compare two different input files, however the examples below will make comparisons between parts of the same input file for demonstration. First, if the two arrays are exactly equal:

from unox.plotting import compare_input_vars

input_comparison = compare_input_vars(
    {
        'input_set': 'no2_2019_JFM',
        'year': 2019,
        'var': 'no2',
    },
    {
        'input_set': 'no2_2019_JFM',
        'year': 2019,
        'var': 'no2',
    }
)

Shape of no2 from no2_2019_JFM: (89, 56, 120)
Shape of no2 from no2_2019_JFM: (89, 56, 120)
Match found for no2_2019_JFM-2019-no2 vs no2_2019_JFM-2019-no2.

If the arrays do have differences, a figure with 6 plots will be made. In these plots, the first array specified will be labeled as “A” and the second, “B”.

from unox.plotting import compare_input_vars

input_comparison = compare_input_vars(
    {
        'input_set': 'no2_2019_JFM',
        'year': 2019,
        'var': 'no2',
    },
    {
        'input_set': 'no2_2019_JFM',
        'year': 2019,
        'var': 'no2_s2',
    },
    # restrict_lat_lon_to='../datafiles/sample_data/nox_2019_t106_US.nc',
)

Shape of no2 from no2_2019_JFM: (89, 56, 120)
Shape of no2_s2 from no2_2019_JFM: (89, 56, 120)
The input files differ more than the tolerance of 2e-05
Number of differences: 15729 / 598080 ( 2.629915730337079 % )

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "

_images/c9e8136bb9b434991e4018d87606e39832a7686cb4f56d9886b8c27b6b1b7b1d.png

Notably, the last subplot, which shows the correlation, only uses the data where the two arrays differ. Below is the correlation plot of the full arrays, where a significant number of data points are along the y=x diagonal.

from unox.plotting import corr_plot

this_plt = corr_plot(
    'no2_2019_JFM',
    is_input_set=True,
    x_vars = 'no2',
    y_vars = 'no2_s2',
)

/home/docs/checkouts/readthedocs.org/user_builds/unox/checkouts/stable/src/unox/HPC/data0/dataset.py:474: UserWarning: (is_ensemble) `dataset` must be a prediction set to check for ensemble members.
  warnings.warn(f"(is_ensemble) `dataset` must be a prediction set to check for ensemble members.")

_images/e2084d77d6c194b523143437b661687e72029e321db11ffb4979f04c0c1583d1.png

back to top

Plotting results

To quickly visualize the results of a model run, use the plot_run_analysis() function. Generally, that would involve a call like the one below.

from unox.plotting import plot_run_analysis

this_plt = plot_run_analysis(
    'no2_example_run',
)

However, the model run no2_example_run, which is included in this repository, was created using an input file called no2_2005-2020.nc. That file is large (3.8 GB) and therefore was not included. If you have followed the Data guide and generated the no2_2005-2020.nc input file in your own instance of the repository, you should be able to copy the above code and run it. For demonstration purposes, the cell below modifies the no2_example_run dataset such that it can act as if it uses the no2_2019_JFM.nc input file. The no2_2019_JFM.nc file, which has been used in above examples, is exactly the same as no2_2005-2020.nc, however has been truncated to only include dates between 2019-01-01 and 2019-03-31 to reduce it’s file size to 60 MB.

from unox.HPC.data0.dataset import uarray 

no2_ex_uarr = uarray('no2_example_run', is_predict=True)
# Modifying the `no2_example_run` dataset such that it can use `no2_2019_JFM` as the input set for plotting functions instead of `no2_2005-2020`, which is too large to include in the repository. This is just for the purpose of demonstrating the plotting functions with a smaller dataset.
no2_ex_uarr.xr.attrs['input_set'] = 'no2_2019_JFM'
no2_ex_uarr._get_metadata()
no2_ex_uarr.metadata['config_dict']['input_set'] = 'no2_2019_JFM'
no2_ex_uarr.xr = no2_ex_uarr.xr.sel(time=slice("2019-01-01", "2019-03-31"))

from unox.plotting import plot_run_analysis

this_plt = plot_run_analysis(
    no2_ex_uarr,
)

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "

_images/385cbac63387219d973429d7fedb79e983d1560fdce2fbce01b580e205dac6ef.png

Similar to the functions shown above, there are many arguments which can be added to adjust this plot.

from unox.plotting import plot_run_analysis

this_plt = plot_run_analysis(
    no2_ex_uarr,
    start_date='2019-01-15',
    interval='2M',
    restrict_lat_lon_to='../datafiles/sample_data/nox_2019_t106_US.nc',
    stage1_only=True,
)

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "

_images/957d3779fe2baac7ff76eb1b64de4cde5931c61e5e7ea0980f23dcd7b82e15c9.png

from unox.plotting import plot_run_analysis

this_plt = plot_run_analysis(
    no2_ex_uarr,
    start_date='2019-01-02',
    interval='20D',
    restrict_lat_lon_to='../datafiles/sample_data/nox_2019_t106_US.nc',
    add_corr_plots=False,
    stage1_only=True,
)

/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "
/home/docs/checkouts/readthedocs.org/user_builds/unox/envs/stable/lib/python3.9/site-packages/cartopy/mpl/geoaxes.py:403: UserWarning: The `map_projection` keyword argument is deprecated, use `projection` to instantiate a GeoAxes instead.
  warnings.warn("The `map_projection` keyword argument is "

_images/8dec1e895d4d3b7d26f2a1495332da77f4a08029c0dcfec8c829f1f8ec3197ae.png

As was mentioned above, the correlation plots can also be made on their own.

from unox.plotting import corr_plot

this_plt = corr_plot(
    no2_ex_uarr,
    is_predict=True,
    start_date='2019-02-01',
    interval='1M',
)

_images/50b061d95d5efd646a29a83ad1bc56b62d95d6008e97237b59fab7667f727771.png

Restricting the latitude and longitude ranges has a noticeable effect on the correlation.

from unox.plotting import corr_plot

this_plt = corr_plot(
    no2_ex_uarr,
    is_predict=True,
    start_date='2019-02-01',
    interval='1M',
    restrict_lat_lon_to='../datafiles/sample_data/nox_2019_t106_US.nc',
)

_images/c1c0528284f9f0814be9a6ce787d66a122960a8bf8f4963ad7c3be4b455d1fb3.png

Several diagnostics are recorded into .csv files during a model run. To view these diagnostics across the epochs of the model run, use the plot_epochs_logs() function.

from unox.plotting import plot_epochs_logs 

this_plot = plot_epochs_logs(
    no2_ex_uarr,
    is_predict=True,
)

_images/f8d6f41197ee71582678b3c658d35cf02163be9041c1160f19380622725645b7.png

Basic Analysis

Table of Contents

Before you start

Exploring a dataset

Inspecting with Jupyter Notebook output

Inspecting with map plots

Inspecting `uarray` objects

Comparing inputs

Plotting results

Basic Analysis

Table of Contents

Before you start

Exploring a dataset

Inspecting with Jupyter Notebook output

Inspecting with map plots

Inspecting uarray objects

Comparing inputs

Plotting results

Inspecting `uarray` objects