Introducing the yt Hub

I am very excited to announce the yt Hub: http://hub.yt-project.org/.

The yt Hub is a gathering place for all things related to running, analyzing, and presenting astrophysical simulations. Once upon a time, we had the Enzotools Barn, which was a place to share Enzo scripts unrelated to yt, including hosting of those scripts. The Barn was a minor success, but had a very limited scope and a somewhat clunky user experience.

Now we have the yt Hub, where users can submit externally- hosted projects in a variety of categories:

Users can create their own accounts and share anything they've created that might be useful to others. Currently, the preferred submission method is to create a repository on bitbucket.org and then submit the link to that. If you would like to host the repository for your contribution somewhere else, that's fine, too. The forthcoming 2.2 release of yt will include a utility for automating the creation and uploading of a bitbucket.org repository as well as submitting that repository to the Hub.

In addition, users may also subscribe to email announcements, in which they will receive an email with any new submissions from the previous day.

Please, come check it out and contribute. The site is brand new, so feel free to send feedback if you encounter any oddities or think of improvements. The primary goal of yt has been to create a common analysis language for computational astrophysicists coming from different simulation codes. We hope that the yt Hub will serve as the meeting place for the conversation.

Author: Britton Smith
Published on: Aug 24, 2011, 6:28:30 PM
Permalink - Source code

We're now the yt Project!

This is just a brief blog entry -- other news is on the way soon! -- to let everyone know we've updated our domains. In keeping with the change in focus of the yt project, expanding to work with multiple codes and take broader views of analysis and visualization, we have moved from enzotools.org to yt-project.org. Some other exciting changes will be coming from this in the future, too!

For more info, see the mailing list announcement.

Author: Matthew Turk
Published on: Aug 22, 2011, 1:39:56 PM
Permalink - Source code

A movie of yt in paraview

Jorge posted this video of using yt inside ParaView. It demonstrated multi-res slices, projections, and even volume rendering with an adjustable transfer function. I think it's pretty incredible. Thanks very much to Jorge and Kitware for their hard work, and it'll be exciting to see where it goes from here!

Author: Matthew Turk
Published on: Aug 9, 2011, 1:49:46 AM
Permalink - Source code

A mission statement and a new homepage!

In case you haven't visited our front page lately, the yt homepage has been redesigned! The old homepage was nice and functional, but it tried to cram too much into too small a place. The new homepage focuses on the three main questions people have when they go to the yt page: 'How do I get it?', 'How do I use it?', and 'How do I participate?'

The new page is focused on some concrete examples, some quantitative analysis, and features a heavy dosage of the yt community -- in fact, the redesign came about through a conversation in IRC, where a few of the core developers were speculating about how best to convey how to participate and how strong a community yt has. The new homepage also features a mission statement. (Discussion of the mission statement.) It's designed to reflect the goals and ambitions -- not where the end of the road is, but which direction it is pointed. To be honest, as the person who initiated that discussion, I was a bit surprised at how far reaching the discussion became. The mission statement we converged on is reproduced here, but over time it might evolve and be refined.

Feel free to stop by the IRC channel or the mailing lists if this piques your interest and you'd like to chat more.

Mission Statement

The yt project aims to produce an integrated science environment for collaboratively asking and answering astrophysical questions. To do so, it will encompass the creation of initial conditions, the execution of simulations, and the detailed exploration and visualization of the resultant data. It will also provide a standard framework based on physical quantities interoperability between codes.

Development of yt is driven by a commitment to Open Science principles as manifested in participatory development, reproducibility, documented and approachable code, a friendly and helpful community of users and developers, and Free and Libre Open Source Software.

Author: Matthew Turk
Published on: Jun 29, 2011, 11:28:05 PM
Permalink - Source code

ParaView and yt

Thanks to some awesome work by developers at Kitware, yt can now be called from within ParaView!

This relies on a couple things, all of which are pretty exciting moving forward. The idea behind this is to leverage where the two codes have their own strengths and weaknesses, and identify places where they can work productively together. For instance, yt has been designed to provide astrophysical analysis; in this way, it can generate adaptive projections, spectral energy distributions and so on. By feeding data to yt (and not asking yt to read any on its own) these functions can be called from within ParaView, and images and results returned back to ParaView.

The strategy taken has, so far, been a light one. Almost all of the yt machinery has been left in place; what happens is a relatively simple process:

We're currently investigating how to streamline this process; currently, it utilizes a new frontend I wrote that was designed to stream data into yt from arbitrary locations. This can act as a proxy for datasets that are already loaded into yt, for datasets that are constructed ab initio in yt and for accepting data through in-memory transfer from ParaView! One of the fun applications of this will be using ParaView as a means for conducting Co-processing of data.

Thanks very much to Jorge, Berk, Charles and George at Kitware for all your hard work to make this happen. This is going to be a really fun way to explore new methods for analysis moving forward.

Not only is this exciting because it enables better cross-talk between yt and ParaView, but also because of the technology that is being developed on both sides. The stream handler in yt, in particular, is a great output from this: it can be used in the future to construct arbitrary datasets in memory, as well as to prototype new frontends for codebases. When yt gains the ability to write out datasets for various code types, this will be a valuable tool for constructing datasets from scratch.

For more information, see this pose on the AMR mailing list at Kitware, which contains an example script. While this functionality will be in the 2.2 release, it will be highlighted in the 2.3 release as we streamline and work out bugs in a proper ParaView frontend.

Author: Matthew Turk
Published on: Jun 14, 2011, 1:09:34 AM
Permalink - Source code

Gmaps-ify your data!

A couple of us have been working on a comprehensive notebook web GUI for yt. It's not ready yet.

BUT! In advance of that, we've rolled a portion of that into something called the 'mapserver' into the development branch. This is a small, standalone webapp that implements a rendered-on-the-fly google maps interface in yt. To run it, just go into a directory that has some data, and run:

yt mapserver DD0054/DD0054

(where 'DD0054/DD0054' is the same thing you'd feed to 'load' in a script.) You can run with --help to see some options, but what it comes down to is that this will slice, but if you want to project, just do -p like so:

yt mapserver -p DD0054/DD0054

This will spawn a webserver on port 8080 which you can then hit in a browser. You'll have to forward an SSH tunnel if you're on a remote machine, but that's just a matter of logging in with ssh -L 8080:localhost:8080. When you're done, just hit Ctrl-C and it'll quit.

/attachments/yt_mapserver.png

Anyway, I think this is pretty cool, and hopefully you will too. If you run into any bugs, report them either with 'yt bugreport' or by going to http://hg.enzotools.org/yt/issues/new .

Author: Matthew Turk
Published on: Jun 10, 2011, 2:08:48 AM
Permalink - Source code

yt Development: Quad trees, Tickets and more

It's been a while since the last Development post — but in that time, some pretty fun things have been going on. These are some of the smaller things, but there are bigger things in store which I'll write about next time.

QuadTree Projections

For a while, there has been the option to use a QuadTree data structure to conduct projections through a simulation volume. This was accessible through the quad_proj attribute, which respected the same interface as the normal proj interface. The old-style projection code required a spatial decomposition in parallel, and did not use the most efficient mechanism for identifying where cells go and which grids overlapped other grids.

The quad tree projection mostly avoided those issues; it could (theoretically) be parallelized with an arbitrary load- balancing scheme, and it should avoid having to calculate any overlaps, as they're inherent to the data structure. However, parallelizing the final combine was never implemented — so while it showed good results for scaling in serial, it didn't work in parallel.

However, as of today, it's now parallel! For big simulations, it should provide a speedup of between 2 and 10 for projecting. For the 2.2 release, this will be the default mechanism for projecting. If you're reading this, you're encouraged to check it out by replacing the old-style projection with the new: pf.h.proj = pf.h.quad_proj

and then conducted any analysis you normally would. Testing and results for scaling would be greatly appreciated!

In-Memory Data Format

If you have some AMR data that you don't want to write a full plugin or frontend for, you're now in luck. One can use the Stream frontend to describe a hierarchy, create a data-reader or data-generator, and then feed this to yt. And example of this can be found here.

The first implementation of this will be for loading data in Paraview and conducting analysis on this data in yt.

Activation Script

We have a new contributor! Casey W. Stark added in activation scripts to the install_script. Using these means you no longer have to set your PYTHONPATH or LD_LIBRARY_PATH or PATH manually, as you can just source the activate scripts. Thanks, Casey!

Tickets! And Bug Reports!

All of the tickets from Trace have been migrated to BitBucket. All bug reports should be left there. Soon the wiki will be moved over, too, and Trac will be mothballed.

Cameron and I are working on a new bug reporting mechanism that can be run from the command line -- yt bugreport -- that should help ensure triaging of bugs occurs in a timely and meaningful fashion.

That's it for this time. Happy simulating!

Author: Matthew Turk
Published on: Jun 3, 2011, 1:20:00 AM
Permalink - Source code

yt Logo Contest, and possibly a free mug!

yt is in need of an awesome new logo, which is why we are announcing the first ever new logo contest! So here's the deal:

We'll accept entries for the next two weeks. Let's arbitrarily say Tuesday, May 10th, at 11:59:59 pm EST. If there is more than one entry, we will put it up for a community vote at that time.

Oh, and the most important part -- the winner will get a coffee mug with their new logo front and center!!!

Please email yt@enzotools.org with your image attached. Good luck, and happy logo-ing!

Sam

Author: Sam Skillman
Published on: Apr 26, 2011, 10:46:57 PM
Permalink - Source code

yt 2.1 Release Announcement

We are proud to announce the release of yt version 2.1. This release includes several new features, bug fixes, and numerous improvements to the code base and documentation. At the yt homepage, http://yt.enzotools.org/ , an installation script, a cookbook, documentation and a guide to getting involved can be found.

yt is an analysis and visualization toolkit for Adaptive Mesh Refinement data. yt provides full support for Enzo, Orion, and FLASH codes, with preliminary support for RAMSES, ART, Chombo, CASTRO and MAESTRO codes. It can be used to create many common types of data products such as:

There are a few major additions since yt-2.0 (Released January 17, 2011), including:

Documentation: http://yt.enzotools.org/doc/

Installation: http://yt.enzotools.org/doc/advanced/installing.html#installing-yt

Cookbook: http://yt.enzotools.org/doc/cookbook/recipes.html

Get Involved: http://yt.enzotools.org/doc/advanced/developing.html#contributing-code

If you can't wait to get started, install with:

$ wget http://hg.enzotools.org/yt/raw/stable/doc/install_script.sh
$ bash install_script.sh

Development has been sponsored by the NSF, DOE, and University funding. We invite you to get involved with developing and using yt!

Author: Sam Skillman
Published on: Apr 8, 2011, 9:57:00 AM
Permalink - Source code

yt Development: Treecodes, GUIs, IRC and more!

It's been nearly a month since the last yt development post; in that time, there's been quite a bit of development in a couple different areas. This is culminating in a 2.1 release, for which Sam Skillman is release manager, sometime in the next few days.

Streamlines and Treecode

SamS has spent some time over the last month developing two types of streamline code. The first integrates a series of streamlines over a selection of the domain, which can then be visualizing using the mplot3d package. The other aspect of this involves selecting one of these integrated streamlines, which is then transformed into an AMR1DData object, which can be queried for values and plotted in other ways. Sam has documented both of these modes of streamline integration and they'll be included in a new build of the documentation presently.

StephenS has been hard at work on developing a treecode for speeding up the binding energy calculation of clumps or other regions. To that end, he's implemented not only the treecode itself (using some of the octree functionality which had as-yet been unused in yt) but also a series of tests. He sees substantial speedups, and it has been documented to become part of the next release.

Development Bootstrap and Pasteboards

The process of getting up and running with Mercurial and with development of yt can be a bit tricky. To try to alleviate that, I've added a new command, "yt bootstrap_dev" that will handle this. It will set up a bitbucket user, set up a couple handly hg extensions, and also create a 'pasteboard.' The pasteboard itself is still a bit in flux, so it's not being touted just yet as a major feature, but I think it has some promise. The idea behind it is to create a semi-permanent mechanism for sharing scripts and so forth; it's a versions hg repository which lives on BitBucket's servers, from which scripts can be programmatically downloaded, embedded, or viewed.

You can see mine at http://matthewturk.bitbucket.org. If you've used the bootstrap_dev command, you can play with the pasteboards with "yt pasteboard" and "yt pastegrab", but be forewarned they might not be completely working yet!

Web GUI and PlotWindow

JeffO has been hard at work rethinking and rebuilding the plotting system in yt. He's started with the concept of the PlotCollection and thrown it out! The PlotCollection dates from a time when our mechanism for interacting with plots was actually fundamentally different; in the first few months of yt's existence, it was operated through a GUI called 'HippoDraw' which operated with worksheets. The mapping was from a single worksheet to a single plot collection.

Nowadays, however, it seems that the most common use of making a plot collection is to make a bunch of plots that aren't quite synced up in the same way that they were with HippoDraw. So Jeff has been rethinking the plotting system, sticking close to the idea of 'conduits' of data that are present in other systems like the AMRData system. You should be able to take a porthole into the data, toss it down, and then simply receive back an image that is visible through that porthole.

Over the last couple weeks he's made quite a bit of progress in that, which enabled us to add it as a widget in a forthcoming GUI for yt. …and, speaking of the GUI, we now have a quite functional protoype of a GUI working. This is the fifth (!) GUI that has been designed for yt. BrittonS, CameronH and JeffO and I took a few days and worked very hard on creating a useful, extensible, and maintainable GUI. (Britton, in fact, had one of the best quotes of the sprint. I said something like, "I'm looking forward to this GUI." Britton replied, "I'm looking forward to this being the last GUI." I couldn't agree with this sentiment more.)

All of the previous versions and implementations of GUIs for yt -- HippoDraw, then the original GUI called Reason and written in wxPython, then the subsequent TraitsUI wxPython Reason version 2.0, and finally the Tkinter-based Fisheye -- were dependent on a whole stack of dependencies. These included things like wx, GTK, Qt, and on and on and on. These were difficult to install in an automated fashion, but more than that, they added an incredible level of complexity to the installation and to using these GUIs on supercomputer centers.

So we decided to get rid of all of that, and return to the basics. We built a Web GUI, where the widgets, the toolkits, events, and everything are all handled by a web browser, using JavaScript. The decision to do this ultimately came down to a maintainability issue -- there's no compilation necessary, everybody has a web browser, and it can be done trivially over SSH (with far less bandwidth than is required for a comparative X11 session being forwarded.)

The underlying system is just a Python interpreter; when buttons are pressed, they simply call functions that are available in the interpreter. It's fundamentally a mechanism for issuing commands, displaying results of those commands (including images), and then on top of this we have begun adding widgets.

Cameron summarized some of this in an email to the developer list. It's still in testing, but is available for testing, despite not being documented. This will be a big part of a 2.2 release of yt.

IRC

There's now an IRC channel for yt, on FreeNode. There aren't usually that many people in there (sometimes it's just me!) but the bot from CIA.vc will echo all pushed commits to the channel. You can use a client like Adium, Irssi, or one of the other Linux or OSX clients, to connect to irc.freenode.net and then to join the channel #yt ("/join #yt").

We'll occasionally have development talk here, but it could also be viewed as a faster turnover mechanism for getting help, chatting about oddities, and suggesting feedback. If you're interested in getting started developing on yt or fixing bugs, this would be the perfect way to get your feet wet. We'll also likely have some coordinated development sessions here in the future.

See you next week!

Author: Matthew Turk
Published on: Apr 4, 2011, 11:40:54 AM
Permalink - Source code