New in 4.1: Loading data with functions
@ Matthew Turk | Thursday, Sep 1, 2022 | 2 minute read | Update at Thursday, Sep 1, 2022

There’s a new feature in yt 4.1 that lets you load data using just functions, without it needing to be in memory or to have its own frontend defined.

Back in the long, long distant past, the Stream frontend was created to make it easy to bring data into yt without writing a full-on “frontend.” Originally this was to make it easy to share data between VTK and yt – and it worked! We were even able to write this up in a blog post over at Kitware’s blog.

The original idea was that if we could separate out the responsibilities for indexing, IO, etc etc, into a set of easily user-definable components, we could make it a lot easier to define individual frontends, including for the ParaView (and VTK) components we used in that post.

That didn’t … exactly happen. Instead, what we ended up doing was writing wrapper functions for some of the most common use cases – load_uniform_grid, load_amr_grids, load_particles, load_hexahedral_mesh, load_unstructured_mesh and load_octree – and then those became the primary drivers of all of the Stream frontend development. They hid almost all of the underlying machinery, and they became where time was invested.

But, that brings me back to a feature I’m really excited about in yt 4.1 – you can now use the Stream frontend to load data that is only accessible via functions. Whereas in the past everything had to be supplied using numpy arrays, now you can just supply function handles. In many ways this is similar to derived fields – those, too, are just functions that are supplied. But the distinction here is that you’re also (by default) supplied spatial information in the function handler, specifically the GridPatch object.

So … what does that mean? Well, it means that if your data is hidden behind some complicated set of HTTP requests, or an API that you haven’t plugged into yt, you can expose it using functions!

This might not sound like a big deal, but I’m pretty excited about what this will open up. One place we’ve seen some real potential for it has been in situations where data is easily accessible from an API, but where the act of writing a frontend poses some overhead.

But this also opens up some pretty neat opportunities to do memory-conservative access of data that is stored in familiar file formats, too!

yt extension modules

yt has many extension packages to help you in your scientific workflow! Check these out, or create your own.

ytini

ytini is set of tools and tutorials for using yt as a tool inside the 3D visual effects software Houdini or a data pre-processor externally to Houdini.

Trident

Trident is a full-featured tool that projects arbitrary sightlines through astrophysical hydrodynamics simulations for generating mock spectral observations of the IGM and CGM.

pyXSIM

pyXSIM is a Python package for simulating X-ray observations from astrophysical sources.

ytree

Analyze merger tree data from multiple sources. It’s yt for merger trees!

yt_idv

yt_idv is a package for interactive volume rendering with yt! It provides interactive visualization using OpenGL for datasets loaded in yt. It is written to provide both scripting and interactive access.

widgyts

widgyts is a jupyter widgets extension for yt, backed by rust/webassembly to allow for browser-based, interactive exploration of data from yt.

yt_astro_analysis

yt_astro_analysis is the yt extension package for astrophysical analysis.

Make your own!!

Finally, check out our development docs on writing your own yt extensions!

Contributing to the Blog

Are you interested in contributing to the yt blog?

Check out our post on contributing to the blog for a guide!

We welcome contributions from all members of the yt community. Feel free to reach out if you need any help.

the yt data hub

The yt hub at https://girder.hub.yt/ has a ton of resources to check out, whether you have yt installed or not.

The collections host all sorts of data that can be loaded with yt. Some have been used in publications, and others are used as sample frontend data for yt. Maybe there’s data from your simulation software?

The rafts host the yt quickstart notebooks, where you can interact with yt in the browser, without needing to install it locally. Check out some of the other rafts too, like the widgyts release notebooks – a demo of the widgyts yt extension pacakge; or the notebooks from the CCA workshop – a user’s workshop on using yt.

Social Links