Quad Tree Projections
@ Matthew Turk | Friday, Sep 10, 2010 | 3 minute read | Update at Friday, Sep 10, 2010

The current method for projections in yt is based on a home-grown algorithm for calculating grid overlap and joining points. I’ve always been pretty proud of it – it gave good results, and it succeeded at the project-once-make-many-images philosophy that went into its design. Rather than storing a 2D array of pixels, projections and slices in yt store flat arrays of image plane coordinates and cell widths. This means that there’s an additional step of pixelization to create an image, but it also means that arbitrary images can be made from a single projection or slice operation.

Over the years, I’ve spent a lot of time shaving time off of the projection mechanism. But, it was still slow on some types of data – and the parallelism was never all that great, either. Because of the algorithm, the parallelism was exclusively based on image-plane decomposition, which means that it was difficult to get good load balancing on nested grid simulations and that it was simply impossible to run inline unless massive amounts of data were to be passed over the wire between processes. This ultimately meant that embedded analysis could not generate adaptive projections, only fixed resolution projections.

A couple months ago, I tried my hand at implementing a new mechanism for projecting based on quad trees. The advantage with Quad Trees would be that they could be decomposed independent of the image plane; this would enable the projections to be made inline, without shipping data around between nodes. The initial results were extremly promising – it was able to project the density field for a 1024^3, 7-level dataset (1.6 billion cells) in just around 2300 seconds in serial, with peak memory usage of 2.7 gigabytes, easily fitting within a single node. I don’t know how long it would take to project this in serial with the old method, because I killed the task after a few hours.

For various reasons, I did not have time to implement this in the main yt code until very recently. It now lives in the main ‘yt’ branch as the AMRQuadProj class, hanging off the hierarchy as .quad_proj. Once it has been parallelized, it will replace the old .proj method, and will be used natively. One of the unfortunate side effects of inserting it into the full yt machinery is that projections have to satisfy a number of API requirements – they have to handle field cuts, arbitrary data objects, and several other reductions that slow things down. However, even with this added baggage, in most cases the new quad tree projection will be about twice as fast as the old method for projections. If you want to give it a go before it becomes the default, you can access it directly by calling .quad_proj on the hierarchy.

Hopefully I’ll be able to parallelize it soon, so that this can become the default method – and so that variable resolution projections can make their way into embedded analysis!

yt extension modules

yt has many extension packages to help you in your scientific workflow! Check these out, or create your own.

ytini

ytini is set of tools and tutorials for using yt as a tool inside the 3D visual effects software Houdini or a data pre-processor externally to Houdini.

Trident

Trident is a full-featured tool that projects arbitrary sightlines through astrophysical hydrodynamics simulations for generating mock spectral observations of the IGM and CGM.

pyXSIM

pyXSIM is a Python package for simulating X-ray observations from astrophysical sources.

ytree

Analyze merger tree data from multiple sources. It’s yt for merger trees!

yt_idv

yt_idv is a package for interactive volume rendering with yt! It provides interactive visualization using OpenGL for datasets loaded in yt. It is written to provide both scripting and interactive access.

widgyts

widgyts is a jupyter widgets extension for yt, backed by rust/webassembly to allow for browser-based, interactive exploration of data from yt.

yt_astro_analysis

yt_astro_analysis is the yt extension package for astrophysical analysis.

Make your own!!

Finally, check out our development docs on writing your own yt extensions!

Contributing to the Blog

Are you interested in contributing to the yt blog?

Check out our post on contributing to the blog for a guide!

We welcome contributions from all members of the yt community. Feel free to reach out if you need any help.

the yt data hub

The yt hub at https://girder.hub.yt/ has a ton of resources to check out, whether you have yt installed or not.

The collections host all sorts of data that can be loaded with yt. Some have been used in publications, and others are used as sample frontend data for yt. Maybe there’s data from your simulation software?

The rafts host the yt quickstart notebooks, where you can interact with yt in the browser, without needing to install it locally. Check out some of the other rafts too, like the widgyts release notebooks – a demo of the widgyts yt extension pacakge; or the notebooks from the CCA workshop – a user’s workshop on using yt.

Social Links