I am very excited to announce the yt Hub: http://hub.yt-project.org/.
The yt Hub is a gathering place for all things related to running, analyzing, and presenting astrophysical simulations. Once upon a time, we had the Enzotools Barn, which was a place to share Enzo scripts unrelated to yt, including hosting of those scripts. The Barn was a minor success, but had a very limited scope and a somewhat clunky user experience.
Now we have the yt Hub, where users can submit externally- hosted projects in a variety of categories:
- yt scripts
- Analysis and visualization tools and utilities
- Simulation and data management
- Scripts for making figures from published papers
- News and announcements
- General astrophysical utilities
Users can create their own accounts and share anything they've created that might be useful to others. Currently, the preferred submission method is to create a repository on bitbucket.org and then submit the link to that. If you would like to host the repository for your contribution somewhere else, that's fine, too. The forthcoming 2.2 release of yt will include a utility for automating the creation and uploading of a bitbucket.org repository as well as submitting that repository to the Hub.
In addition, users may also subscribe to email announcements, in which they will receive an email with any new submissions from the previous day.
Please, come check it out and contribute. The site is brand new, so feel free to send feedback if you encounter any oddities or think of improvements. The primary goal of yt has been to create a common analysis language for computational astrophysicists coming from different simulation codes. We hope that the yt Hub will serve as the meeting place for the conversation.
This is just a brief blog entry -- other news is on the way soon! -- to let everyone know we've updated our domains. In keeping with the change in focus of the yt project, expanding to work with multiple codes and take broader views of analysis and visualization, we have moved from enzotools.org to yt-project.org. Some other exciting changes will be coming from this in the future, too!
For more info, see the mailing list announcement.
Jorge posted this video of using yt inside ParaView. It demonstrated multi-res slices, projections, and even volume rendering with an adjustable transfer function. I think it's pretty incredible. Thanks very much to Jorge and Kitware for their hard work, and it'll be exciting to see where it goes from here!
In case you haven't visited our front page lately, the yt homepage has been redesigned! The old homepage was nice and functional, but it tried to cram too much into too small a place. The new homepage focuses on the three main questions people have when they go to the yt page: 'How do I get it?', 'How do I use it?', and 'How do I participate?'
The new page is focused on some concrete examples, some quantitative analysis, and features a heavy dosage of the yt community -- in fact, the redesign came about through a conversation in IRC, where a few of the core developers were speculating about how best to convey how to participate and how strong a community yt has. The new homepage also features a mission statement. (Discussion of the mission statement.) It's designed to reflect the goals and ambitions -- not where the end of the road is, but which direction it is pointed. To be honest, as the person who initiated that discussion, I was a bit surprised at how far reaching the discussion became. The mission statement we converged on is reproduced here, but over time it might evolve and be refined.
Feel free to stop by the IRC channel or the mailing lists if this piques your interest and you'd like to chat more.
The yt project aims to produce an integrated science environment for collaboratively asking and answering astrophysical questions. To do so, it will encompass the creation of initial conditions, the execution of simulations, and the detailed exploration and visualization of the resultant data. It will also provide a standard framework based on physical quantities interoperability between codes.
Development of yt is driven by a commitment to Open Science principles as manifested in participatory development, reproducibility, documented and approachable code, a friendly and helpful community of users and developers, and Free and Libre Open Source Software.
Thanks to some awesome work by developers at Kitware, yt can now be called from within ParaView!
This relies on a couple things, all of which are pretty exciting moving forward. The idea behind this is to leverage where the two codes have their own strengths and weaknesses, and identify places where they can work productively together. For instance, yt has been designed to provide astrophysical analysis; in this way, it can generate adaptive projections, spectral energy distributions and so on. By feeding data to yt (and not asking yt to read any on its own) these functions can be called from within ParaView, and images and results returned back to ParaView.
The strategy taken has, so far, been a light one. Almost all of the yt machinery has been left in place; what happens is a relatively simple process:
- Load data into ParaView
- Feed data into yt
- Construct IO handlers that pass the contents of internal VTK structures to yt through a custom IOHandlerBase implementation.
- Construct a hierarchy from the existing contents of the vtkHierarchicalBoxDataSet object, and its attendant child objects.
- Execute analysis
- Return processed data to paraview
We're currently investigating how to streamline this process; currently, it utilizes a new frontend I wrote that was designed to stream data into yt from arbitrary locations. This can act as a proxy for datasets that are already loaded into yt, for datasets that are constructed ab initio in yt and for accepting data through in-memory transfer from ParaView! One of the fun applications of this will be using ParaView as a means for conducting Co-processing of data.
Thanks very much to Jorge, Berk, Charles and George at Kitware for all your hard work to make this happen. This is going to be a really fun way to explore new methods for analysis moving forward.
Not only is this exciting because it enables better cross-talk between yt and ParaView, but also because of the technology that is being developed on both sides. The stream handler in yt, in particular, is a great output from this: it can be used in the future to construct arbitrary datasets in memory, as well as to prototype new frontends for codebases. When yt gains the ability to write out datasets for various code types, this will be a valuable tool for constructing datasets from scratch.
For more information, see this pose on the AMR mailing list at Kitware, which contains an example script. While this functionality will be in the 2.2 release, it will be highlighted in the 2.3 release as we streamline and work out bugs in a proper ParaView frontend.
A couple of us have been working on a comprehensive notebook web GUI for yt. It's not ready yet.
BUT! In advance of that, we've rolled a portion of that into something called the 'mapserver' into the development branch. This is a small, standalone webapp that implements a rendered-on-the-fly google maps interface in yt. To run it, just go into a directory that has some data, and run:
yt mapserver DD0054/DD0054
(where 'DD0054/DD0054' is the same thing you'd feed to 'load' in a script.) You can run with --help to see some options, but what it comes down to is that this will slice, but if you want to project, just do -p like so:
yt mapserver -p DD0054/DD0054
This will spawn a webserver on port 8080 which you can then hit in a browser. You'll have to forward an SSH tunnel if you're on a remote machine, but that's just a matter of logging in with ssh -L 8080:localhost:8080. When you're done, just hit Ctrl-C and it'll quit.
Anyway, I think this is pretty cool, and hopefully you will too. If you run into any bugs, report them either with 'yt bugreport' or by going to http://hg.enzotools.org/yt/issues/new .
It's been a while since the last Development post — but in that time, some pretty fun things have been going on. These are some of the smaller things, but there are bigger things in store which I'll write about next time.
For a while, there has been the option to use a QuadTree data structure to conduct projections through a simulation volume. This was accessible through the quad_proj attribute, which respected the same interface as the normal proj interface. The old-style projection code required a spatial decomposition in parallel, and did not use the most efficient mechanism for identifying where cells go and which grids overlapped other grids.
The quad tree projection mostly avoided those issues; it could (theoretically) be parallelized with an arbitrary load- balancing scheme, and it should avoid having to calculate any overlaps, as they're inherent to the data structure. However, parallelizing the final combine was never implemented — so while it showed good results for scaling in serial, it didn't work in parallel.
However, as of today, it's now parallel! For big simulations, it should provide a speedup of between 2 and 10 for projecting. For the 2.2 release, this will be the default mechanism for projecting. If you're reading this, you're encouraged to check it out by replacing the old-style projection with the new: pf.h.proj = pf.h.quad_proj
and then conducted any analysis you normally would. Testing and results for scaling would be greatly appreciated!
If you have some AMR data that you don't want to write a full plugin or frontend for, you're now in luck. One can use the Stream frontend to describe a hierarchy, create a data-reader or data-generator, and then feed this to yt. And example of this can be found here.
The first implementation of this will be for loading data in Paraview and conducting analysis on this data in yt.
We have a new contributor! Casey W. Stark added in activation scripts to the install_script. Using these means you no longer have to set your PYTHONPATH or LD_LIBRARY_PATH or PATH manually, as you can just source the activate scripts. Thanks, Casey!
All of the tickets from Trace have been migrated to BitBucket. All bug reports should be left there. Soon the wiki will be moved over, too, and Trac will be mothballed.
Cameron and I are working on a new bug reporting mechanism that can be run from the command line -- yt bugreport -- that should help ensure triaging of bugs occurs in a timely and meaningful fashion.
That's it for this time. Happy simulating!
yt is in need of an awesome new logo, which is why we are announcing the first ever new logo contest! So here's the deal:
We'll accept entries for the next two weeks. Let's arbitrarily say Tuesday, May 10th, at 11:59:59 pm EST. If there is more than one entry, we will put it up for a community vote at that time.
Oh, and the most important part -- the winner will get a coffee mug with their new logo front and center!!!
Please email email@example.com with your image attached. Good luck, and happy logo-ing!
We are proud to announce the release of yt version 2.1. This release includes several new features, bug fixes, and numerous improvements to the code base and documentation. At the yt homepage, http://yt.enzotools.org/ , an installation script, a cookbook, documentation and a guide to getting involved can be found.
yt is an analysis and visualization toolkit for Adaptive Mesh Refinement data. yt provides full support for Enzo, Orion, and FLASH codes, with preliminary support for RAMSES, ART, Chombo, CASTRO and MAESTRO codes. It can be used to create many common types of data products such as:
- Arbitrary Data Selection
- Cosmological Analysis
- Halo finding
- Parallel AMR Volume Rendering
- Gravitationally Bound Objects Analysis
There are a few major additions since yt-2.0 (Released January 17, 2011), including:
- Streamlines for visualization and querying
- A treecode implementation to calculate binding energy
- Healpix / all-sky parallel volume rendering
- A development bootstrap script, for getting going with modifying and contributing
- CASTRO particles
- Time series analysis
If you can't wait to get started, install with:
$ wget http://hg.enzotools.org/yt/raw/stable/doc/install_script.sh $ bash install_script.sh
Development has been sponsored by the NSF, DOE, and University funding. We invite you to get involved with developing and using yt!
It's been nearly a month since the last yt development post; in that time, there's been quite a bit of development in a couple different areas. This is culminating in a 2.1 release, for which Sam Skillman is release manager, sometime in the next few days.
SamS has spent some time over the last month developing two types of streamline code. The first integrates a series of streamlines over a selection of the domain, which can then be visualizing using the mplot3d package. The other aspect of this involves selecting one of these integrated streamlines, which is then transformed into an AMR1DData object, which can be queried for values and plotted in other ways. Sam has documented both of these modes of streamline integration and they'll be included in a new build of the documentation presently.
StephenS has been hard at work on developing a treecode for speeding up the binding energy calculation of clumps or other regions. To that end, he's implemented not only the treecode itself (using some of the octree functionality which had as-yet been unused in yt) but also a series of tests. He sees substantial speedups, and it has been documented to become part of the next release.
The process of getting up and running with Mercurial and with development of yt can be a bit tricky. To try to alleviate that, I've added a new command, "yt bootstrap_dev" that will handle this. It will set up a bitbucket user, set up a couple handly hg extensions, and also create a 'pasteboard.' The pasteboard itself is still a bit in flux, so it's not being touted just yet as a major feature, but I think it has some promise. The idea behind it is to create a semi-permanent mechanism for sharing scripts and so forth; it's a versions hg repository which lives on BitBucket's servers, from which scripts can be programmatically downloaded, embedded, or viewed.
You can see mine at http://matthewturk.bitbucket.org. If you've used the bootstrap_dev command, you can play with the pasteboards with "yt pasteboard" and "yt pastegrab", but be forewarned they might not be completely working yet!
JeffO has been hard at work rethinking and rebuilding the plotting system in yt. He's started with the concept of the PlotCollection and thrown it out! The PlotCollection dates from a time when our mechanism for interacting with plots was actually fundamentally different; in the first few months of yt's existence, it was operated through a GUI called 'HippoDraw' which operated with worksheets. The mapping was from a single worksheet to a single plot collection.
Nowadays, however, it seems that the most common use of making a plot collection is to make a bunch of plots that aren't quite synced up in the same way that they were with HippoDraw. So Jeff has been rethinking the plotting system, sticking close to the idea of 'conduits' of data that are present in other systems like the AMRData system. You should be able to take a porthole into the data, toss it down, and then simply receive back an image that is visible through that porthole.
Over the last couple weeks he's made quite a bit of progress in that, which enabled us to add it as a widget in a forthcoming GUI for yt. …and, speaking of the GUI, we now have a quite functional protoype of a GUI working. This is the fifth (!) GUI that has been designed for yt. BrittonS, CameronH and JeffO and I took a few days and worked very hard on creating a useful, extensible, and maintainable GUI. (Britton, in fact, had one of the best quotes of the sprint. I said something like, "I'm looking forward to this GUI." Britton replied, "I'm looking forward to this being the last GUI." I couldn't agree with this sentiment more.)
All of the previous versions and implementations of GUIs for yt -- HippoDraw, then the original GUI called Reason and written in wxPython, then the subsequent TraitsUI wxPython Reason version 2.0, and finally the Tkinter-based Fisheye -- were dependent on a whole stack of dependencies. These included things like wx, GTK, Qt, and on and on and on. These were difficult to install in an automated fashion, but more than that, they added an incredible level of complexity to the installation and to using these GUIs on supercomputer centers.
The underlying system is just a Python interpreter; when buttons are pressed, they simply call functions that are available in the interpreter. It's fundamentally a mechanism for issuing commands, displaying results of those commands (including images), and then on top of this we have begun adding widgets.
Cameron summarized some of this in an email to the developer list. It's still in testing, but is available for testing, despite not being documented. This will be a big part of a 2.2 release of yt.
There's now an IRC channel for yt, on FreeNode. There aren't usually that many people in there (sometimes it's just me!) but the bot from CIA.vc will echo all pushed commits to the channel. You can use a client like Adium, Irssi, or one of the other Linux or OSX clients, to connect to irc.freenode.net and then to join the channel #yt ("/join #yt").
We'll occasionally have development talk here, but it could also be viewed as a faster turnover mechanism for getting help, chatting about oddities, and suggesting feedback. If you're interested in getting started developing on yt or fixing bugs, this would be the perfect way to get your feet wet. We'll also likely have some coordinated development sessions here in the future.
See you next week!