A few of us worked this past week on a couple yt projects and made what we think is significant progress. Two of the items we focused on were testing and parallelism.
For testing, we’ve broadened the test suite to include many more functions and derived quantities. We now have 548 tests that include (off and on-axis) slices, (off and on- axis) projections, phase distributions, halo finding, volume rendering, and geometrical region cuts such as rectangular solids, spheres, and disks.
This last week, following the release of version 2.2 of yt, I spent a bit of time looking at speed improvements. There were several places that the code was unacceptably slow:
1D profiles (as noted in our method paper, even) Ghost-zone generation RAMSES grid data loading The first of these was relatively easy to fix. In the past, 1D profiles (unlike 2D profiles) were calculated using pure-python mechanisms; numpy was used for digitization, then inverse binning was conducted by the numpy ‘where’ command, and these binnings were used to generate the overall histogram.
It’s been a while since the last Development post — but in that time, some pretty fun things have been going on. These are some of the smaller things, but there are bigger things in store which I’ll write about next time.
QuadTree Projections For a while, there has been the option to use a QuadTree data structure to conduct projections through a simulation volume. This was accessible through the quad_proj attribute, which respected the same interface as the normal proj interface.
The major changes this week came mostly in the form of administrative shifts. However, SamS did some great work I’m going to hint at (he’ll post a blog entry later) and I started laying the ground work for something I’ve been excited about for a while, an MPI-aware task queue.
BitBucket For the last couple months, yt has been struggling under the constraints of the hg server on its hosting plan.
Not much yt development went on in the last week; I spent some time working with Enzo and driving forward simulation goals, which resulted in some development that directly benefited those simulation goals. However, this fortuitously coincided with work I have been eager to return to for quite some time: namely, time series analysis!
Time Series Analysis The problem with time series analysis in yt has, to this point, been an issue of verbosity and clunkiness.
This is the second blog entry in the weekly series, with some updates on what took place last week with respect to yt development. One of the more exciting things is the final one, which is the start of what I want to focus on for the next couple months or years: integration of physics modules with analysis code, and then the ultimate inversion of that relationship.
yt-2.0 This week saw the release of yt 2.
This is the first of a new series of “what’s up with yt” blog posts I’m going to be writing. By keeping this log, I hope that maybe some things that would otherwise get lost in the version control changesets will get brought to greater light. This covers the time period of the first couple weeks in January.
Star Particle Rendering On the mailing list, the question of adding star particles to a volume rendering was raised.
After a significant amount of development and restructuring, I have added the AMR kd-Tree rendering framework to yt. There are several posts on this blog about this module already, so I won’t go over all the background information again. Here I’d like to showcase some of the recent successes and capabilities of the volume rendering within yt.
New optimization options:There are a few important additions that have made it possible to render some of the largest AMR simulations we have available.
Just sharing a video here that showcases some improvements I’ve made to the kD-tree rendering that will be making its way to yt for the 2.0 release. You can download it render_movie.3gp|here
Just to be clear this is showing the rendering of a cosmology simulation with a 64^3 root grid + 6 AMR levels in real time on 8 processors. The script is run in parallel, with the root processor displaying the results once each frame is finished.
The current method for projections in yt is based on a home-grown algorithm for calculating grid overlap and joining points. I’ve always been pretty proud of it – it gave good results, and it succeeded at the project-once-make-many-images philosophy that went into its design. Rather than storing a 2D array of pixels, projections and slices in yt store flat arrays of image plane coordinates and cell widths. This means that there’s an additional step of pixelization to create an image, but it also means that arbitrary images can be made from a single projection or slice operation.
The Parallel Merger Tree in yt, like most of the code in yt, has a rich history of changes and upgrades. One of the most significant upgrades was a change in the way the SQLite database file is updated during the course of building the merger tree. Briefly, the database contains all the information about the merger tree of the halos, as well as the specifics of each halo, such as the mass, position or bulk velocity.
yt has many extension packages to help you in your scientific workflow! Check
these out, or create your own.
ytini is set of tools and tutorials for using yt as a tool inside the 3D visual
effects software Houdini or a data pre-processor externally to Houdini.
Trident is a full-featured tool that projects arbitrary sightlines through
astrophysical hydrodynamics simulations for generating mock spectral
observations of the IGM and CGM.
pyXSIM is a Python package for simulating X-ray observations from astrophysical
Analyze merger tree data from multiple sources. It’s yt for merger trees!
yt_idv is a package for interactive volume rendering with yt!
It provides interactive visualization using OpenGL for datasets
loaded in yt. It is written to provide both scripting and interactive access.
widgyts is a jupyter widgets extension for yt,
backed by rust/webassembly to allow for browser-based, interactive exploration
of data from yt.
We welcome contributions from all members of the yt community. Feel free to
reach out if you need any help.
the yt data hub
The yt hub at https://girder.hub.yt/ has a ton of resources to check out,
whether you have yt installed or not.
The collections host all sorts of data
that can be loaded with yt. Some have been used in publications, and others are
used as sample frontend data for yt. Maybe there’s data from your simulation
The rafts host the yt quickstart notebooks,
where you can interact with yt in the browser, without needing to install it
locally. Check out some of the other rafts too, like the widgyts release
notebooks – a demo of the widgyts yt extension pacakge; or the notebooks from
the CCA workshop – a user’s workshop on using yt.