yt Project Blog http://blog.yt-project.org Most recent posts at yt Project Blog posterous.com Mon, 30 Apr 2012 05:42:04 -0700 yt Google+ Hangout tomorrow! http://blog.yt-project.org/yt-google-hangout-tomorrow http://blog.yt-project.org/yt-google-hangout-tomorrow Tomorrow we're going to try something new -- Google Hangouts! If you'd like help with something, to share some feedback, or just to say hi to other community members, stop by Tuesday, May 1st. We'll be starting up around 2PM Eastern and continuing for a couple hours.

If this works out, we'll try it again from time to time, to catch up on new developments, help out with scripts or visualization issues, soliciting feedback, and to chat about using and developing yt.

You'll find the Hangout on the yt Google Plus page.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Mon, 13 Feb 2012 04:10:21 -0800 What's new with yt? http://blog.yt-project.org/whats-new-with-yt http://blog.yt-project.org/whats-new-with-yt Now that the post-workshop preparations and work have settled down, I thought it might be interesting to share some of the developments going on with yt.  We're still a long way from a new release, so these interim "development" updates are meant to be a bit of a teaser.  As always, these features are either in the main branch or (if noted) in a public fork on BitBucket.  If they sound interesting, drop us a line on yt-dev to see about getting involved!

Stephen has been pushing lately for more consistency in the code base -- indentation, naming conventions, spaces, and so on.  Specifically, he has been suggesting we follow PEP-8, which is a standard for Python source code.  This has gotten a lot of support, and so we're encouraging this in new commits and looking into mechanisms for updating old code.  (Although it can cause some tricky merges, so we're trying to take it easy for a bit!)

JohnZ recently added a particle trajectory mechanism, for correlating particles between outputs and following them.  This lets you see where they go and the character of the gas they pass through.

Sam has been looking at improving volume rendering, including adding hard surfaces and a much faster (Cythonized) kD-tree routine.  The initial hard surface stuff looks just great.  (This is all taking place in his fork.)  This code is also threaded, so it should run much faster on multi-core machines.

JohnW identified a bug in the ghost zone generation, which has resulted in a big speedup for generating ghost zones!

Chris has been trying to get the regridding process for ART to be substantially faster, which he's been having success with.  We're now trying to together work on changing how "child masking" is thought of; with patch-based codes it only masks those cells where data at finer levels is available.  We're trying to make it so that it also marks where coarser data is the finest available, which should help out with speed for octree based codes.

Finally, I've been up to working on geometric selection.  My hope is that by rethinking how we think about geometry in yt and removing a number of intermediate steps, we can avoid creating a whole bunch of temporary arrays and overall speed up the process (and add better support for non-patch based codes!).  Results so far have been pretty good, but it's a long ways from being ready.  It's in my refactor fork.

There are a lot of exciting things going on, so keep your eyes on this space!  In addition to all of these things, we've got web interactors for isolated routines, an all-new hub, improvements to reason, and tons of other stuff.  As always, drop by yt-dev or the IRC channel if you'd like to get involved.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Mon, 30 Jan 2012 10:02:09 -0800 yt workshop 2012: a success! http://blog.yt-project.org/yt-workshop-2012-a-success http://blog.yt-project.org/yt-workshop-2012-a-success The yt workshop last week in Chicago (
http://yt-project.org/workshop2012/ ) was an enormous success. On
behalf of the organizing and technical committees, I'd like to
specifically thank the FLASH Center, particularly Don Lamb, Mila
Kuntu, Carrie Eder, for their hospitality; the venue was outstanding
and their hospitality touching. Additionally, we're very grateful to
the Adler Planetarium's Doug Roberts and Mark SubbaRao for hosting us
on Wednesday evening -- seeing the planetarium show as well as volume
renderings made by yt users up on the dome was so much fun. The yt
workshop was supported by NSF Grant 1214147. Thanks to everyone who
attended -- your energy and excitement helped make it a success.

Thanks also to the organizing and technical committees: Britton
Smith, John ZuHone, Brian O'Shea, Jeff Oishi, Stephen Skory, Sam
Skillman, and Cameron Hummels. All talks have been recorded, and you
can clone a unified repository of talk slides and worked examples:

hg clone https://bitbucket.org/yt_analysis/workshop2012/

A few photos have been put up online, too:

http://goo.gl/g02uP

As I am able to edit and upload talks, they'll appear on the yt
youtube channel as well as on the yt homepage:

http://www.youtube.com/ytanalysis

Thanks again, and wow, what a week!

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Mon, 16 Jan 2012 19:45:43 -0800 Workshop in just a week! http://blog.yt-project.org/workshop-in-just-a-week http://blog.yt-project.org/workshop-in-just-a-week The first yt workshop is in just about a week.  We've updated the website with the current list of talks, along with information about getting to and from the workshop from the conference hotel, and information about how to get the sample data.  Keep your eyes on the website in the lead up to the workshop, as we'll be posting a script for fisheye lens renderings for our viz night at the Adler, information about the talks and example scripts, and other useful info.  Once the workshop is over we'll update with links to the full videos of the talks, the slides, and scripts.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Thu, 15 Dec 2011 11:44:00 -0800 yt Version 2.3 Announcement http://blog.yt-project.org/yt-version-23-announcement http://blog.yt-project.org/yt-version-23-announcement

Just in time for the New Year, we’re happy to announce the release of yt version 2.3!  ( http://yt-project.org/ )  The new version includes many new modules and enhancements, and the usual set of bug fixes over the last point release. We encourage all users to upgrade to take advantage of the changes.

yt is a community-developed analysis and visualization toolkit for astrophysical simulation data. yt provides full support for Enzo, Orion, Nyx, and FLASH codes, with preliminary support for the RAMSES code (and a handful of others.)  It can be used to create many common types of data products, as well as serving as a library for developing your own data reductions and processes.

Below is a non-comprehensive list of new features and enhancements:

 

  • Improved and expanded documentation located at http://yt-project.org/doc/.
  • Boolean logic data containers (joins, intersections and nots) to select arbitrary data regions.
  • Multi-level parallelism for subgroups of MPI tasks.
  • Extensive answer tests.
  • Isocontouring and flux-over-surface calculations, with WebGL interface.
  • A reorganized field system.
  • Adaptive resolution HEALpix-based all-sky volume rendering.
  • Radial column density calculations.
  • Memory usage, performance enhancements and bug fixes throughout the code.

 

Everything, from installation, to development, to a cookbook, can be found on the homepage: http://yt-project.org/

We have updated the libraries installed with the install script; for more information, see the “Dependencies” section of the yt docs at http://yt-project.org/doc/advanced/installing.html.

Development has been sponsored by the NSF, DOE, and various University funding. We invite you to get involved with developing and using yt!

We’re also holding the FIRST YT WORKSHOP from January 24-26 at the FLASH center in Chicago.  See the workshop homepage for more information!  http://yt-project.org/workshop2012/

Please forward this announcement to interested parties.

Sincerely,

    The yt development team

 

Permalink | Leave a comment  »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/heOzHGBSGnjGG Stephen Skory Stephen Stephen Skory Mon, 28 Nov 2011 14:17:49 -0800 yt workshop registration is open! http://blog.yt-project.org/yt-workshop-registration-is-open http://blog.yt-project.org/yt-workshop-registration-is-open Registration for the 2012 yt workshop, to be held at the FLASH center
in Chicago from January 24-26, is now open.

Please register here: http://yt-project.org/workshop2012/

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Wed, 09 Nov 2011 08:03:00 -0800 Boolean Data Containers http://blog.yt-project.org/boolean-data-containers http://blog.yt-project.org/boolean-data-containers

A useful new addition to yt are boolean data containers. These are hybrid data containers that are built by relating already-defined data containers with each other using boolean operators. Nested boolean logic, using parentheses, is also supported. The boolean data container (or volume) is made by constructing a list of volumes interspersed with operators given as strings. Below are some examples of what can be done with boolean data containers.

The “OR” Operator

The “OR” operator combines volume of the two data containers into one. The two intial volumes may or may not overlap, meaning that the combined volume may constitute several disjoint volumes. Here is an example showing the construction of a boolean volume of two disjoint spheres:

sp1 = pf.h.sphere([0.3]*3, .15)
sp2 = pf.h.sphere([0.7]*3, .25)
bool = pf.h.boolean([sp1, "OR", sp2])

Here is a short video showing the result:

The “AND” Operator

The “AND” operator mixes two volumes where both volumes cover the same volume. Put another way, the “AND” operator produces a new volume that is defined by all cells that lie in both of the initial volumes. Here is an example of the intersection of a sphere and a cube:

re1 = pf.h.region([0.5]*3, [0.0]*3, [0.7]*3)
sp1 = pf.h.sphere([0.5]*3, 0.5)
bool = pf.h.boolean([re1, "AND", sp1])

Here is a short video showing the result:

The “NOT” Operator

The “NOT” operator is the only non-transitive operator, and is read from left to right. For example, if there are multiple “NOT” operators, the first “NOT” on the left and the two volumes on either side are considered first. The new volume constructed is the volume contained in the first data container that the second data container does not cover. This can be thought of as a subtraction from the first volume by the second volume. Here is an example of a cubical region having a corner cut out of it:

re1 = pf.h.region([0.5]*3, [0.]*3, [1.]*3)
re2 = pf.h.region([0.5]*3, [0.5]*3, [1.]*3)
bool = pf.h.boolean([re1, "NOT", re2])

Here is a short video showing the result:

Nested Logic

It is possible to use nested logic using parentheses. When nested logic is used, the order of logical operations begins at the inner-most nested level and proceeds outwards, always respecting the left to right ordering for “NOT” operations. This may be used to create truly fantastic volumes. Here is an example of a piece of Swiss cheese created from two cubical regions and two spheres. The second sphere sp2 wraps around the periodic boundaries and impacts the largest cube in more than one place.

re1 = pf.h.region([0.5]*3, [0.]*3, [1.]*3)
re2 = pf.h.region([0.5]*3, [0.5]*3, [1.]*3)
sp1 = pf.h.sphere([0.5, 0.7, 0.5], .25)
sp2 = pf.h.sphere([0.1]*3, .25)
bool = pf.h.boolean([re1, "NOT", "(", re2, "AND", sp1, ")", "NOT", sp2])

Here is a short video showing the result:

For those wondering how the movies were made, I’ve posted the script here. Note that blocks of comments will need to be turned on/off to get the desired boolean data container.

Permalink | Leave a comment  »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/heOzHGBSGnjGG Stephen Skory Stephen Stephen Skory Wed, 09 Nov 2011 04:23:00 -0800 Google+ Page for yt http://blog.yt-project.org/google-page-for-yt http://blog.yt-project.org/google-page-for-yt yt now has a Google Plus page.  Here's we'll post smaller, less blog-worthy items, hold video conferencing "hangouts", and so on.  Encircle away!  And if you post something you'd like to be reshared, just be sure to explicitly share it with "+yt" so we know.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Mon, 24 Oct 2011 08:14:22 -0700 yt Testing & Parallelism http://blog.yt-project.org/yt-testing-parallelism http://blog.yt-project.org/yt-testing-parallelism

Hi all,

A few of us worked this past week on a couple yt projects and made what we think is significant progress.  Two of the items we focused on were testing and parallelism.    

For testing, we've broadened the test suite to include many more functions and derived quantities.  We now have 548 tests that include (off and on-axis) slices, (off and on-axis) projections, phase distributions, halo finding, volume rendering, and geometrical region cuts such as rectangular solids, spheres, and disks.  We use both plain and derived fields for these tests so that it covers as many bases as possible.  With this framework, we are now able to keep a gold standard of the test results for any dataset, then test later changes against this standard.  These tests can test for bitwise identicality or allow for some tolerance. For a full list of tests, you can run python yt/tests/runall.py -l, and use --help to look at the usage.  We will soon be updating the documentation to provide more information on how to set up the testing framework, but I think all of us agree that this will make it much easier to test our changes to make sure bugs have not crept in.

The second big change I'd like to talk about is the way we now handle parallelism in yt.  Previously, methods that employed parallelism through MPI calls would first inherit from ParallelAnalysisInterface, which had access to a ton of mpi functions that all work off of MPI.COMM_WORLD.  In our revamp we wanted to accomplish two things: 1) merge duplicate mpi calls that were only different by the type of values they work on and do overall cleanup. 2) Allow for nested levels of parallelism where two (or more) separate communicators are able to use barriers and collective operations such as allreduce.  To do this, we worked in a two-step process.  First we took things like:

    def _mpi_allsum(self, data):
    def _mpi_Allsum_double(self, data):
    def _mpi_Allsum_long(self, data):
    def _mpi_allmax(self, data):
    def _mpi_allmin(self, data):

and packed it into a single function:

    def mpi_allreduce(self, data, dtype=None, op='sum'):

When a numpy array is passed to this new mpi_allreduce, dtype is determined from the array properties.  If the data is a dictionary, then it is passed to mpi4py's allreduce function that acts on dictionaries.  This greatly reduced the number of lines in parallel_analysis_interface (1376 ==> 915), even after adding in additional functionality.

The second step was bundling all of these functions into a new class called Communicator.  This Communicator object is initialized with an MPI communicator that no longer is restricted to COMM_WORLD.  Using this as the fundamental MPI object, we then built a CommunicationSystem object that manages these communicators.  A global communication_system instance is created, that is initialized with COMM_WORLD at the top of the system if the environment is mpi4py-capable.  If not, an empty communicator is created that has passthroughs for all the mpi functions.

Using this new framework we are now able to take advantage of multiple communicators.  There are two use cases that we have implemented so far: 
1) parallel_objects
parallel_objects is a method un parallel_analysis_interface.py for iterating over a set of objects such that a group of processors work on each object.  This could be used, for example, to run N projections each with M processors, allowing for a parallelism of NxM.  

2) workgroups
workgoups allows users to set up multiple MPI communicators with a non-uniform number of processors to each work on a separate task.  This capability lives within the ProcessorPool and Workgroup objects in parallel_analysis_interface.py

These are just the first two that we tried out and we are very excited about the new possibilities.  

With these changes, there was one implementation change that has already come up once in the mailing list.  When you implement a new class that you'd like to have access to the communication objects, you must first inherit ParallelAnalysisInterface, and then make sure that __init__ makes a call to:
ParallelAnalysisInterface.__init__()

At that point, your new class will have access to the mpi calls through the self.comm object.  For example, to perform a reduction one would do:
self.comm.mpi_allreduce(my_data, op='sum')

As I said before, this will be documented soon, but hopefully this will help for now.

Sam, Britton, Cameron, Jeff, and Matt

 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/548841/Sam_Skillman.png http://posterous.com/users/5AvFMjXVEE7v Sam Skillman Sam Sam Skillman Mon, 03 Oct 2011 18:24:29 -0700 Announcing the 2012 yt Workshop http://blog.yt-project.org/announcing-the-2012-yt-workshop http://blog.yt-project.org/announcing-the-2012-yt-workshop I'm pleased to announce the 2012 yt Workshop at the FLASH Center in Chicago, January 24-26.

The workshop will be aimed at both users and developers of yt.  We will begin with intensive user training, moving from basic usage to advanced and parallel usage.  Users are encouraged to bring their ideas and prototypes for new analysis routines as there will be opportunities to work with more experienced developers.  We will then address to how to modify, extend and contribute to yt, and transition to a developers workshop.  In the developers portion of the workshop, we will discuss ideas for improvements to the code and then break into groups to implement new features.  Users are highly encouraged to stay and participate in development.

The FLASH Center has graciously offered to host the workshop.  We have identified a hotel in downtown Chicago (near the river, just off Michigan Avenue) that we are able to book double-occupancy rooms for $99/night, pre-tax.  We are actively pursuing funding opportunities, but as of yet have not secured funding for participant costs; if we are able to do so, it will likely cover hotel stays for a limited number of individuals willing to share rooms for the four nights of the workshop (Jan. 23-26).

As we prepare hotel reservations, funding applications and other technicalities, we need to get a sense of not only how many people are potentially going to attend, but also their current career stage, funding availability and so on. If you are interested in attending the workshop, we would greatly appreciate it if you would visit the following URL and fill out the Google Form: http://goo.gl/xElrB .  If you have already filled it out, no need to do it again!

Once we have the details of the conference settled, further information will be forthcoming regarding registration, accommodations, and possible financial support.

For specific questions regarding the workshop, please email John ZuHone at jzuhone [at] milkyway.gsfc.nasa.gov.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Tue, 27 Sep 2011 13:20:56 -0700 New Screencasts http://blog.yt-project.org/new-screencasts http://blog.yt-project.org/new-screencasts A few of the yt developers have been experimenting with screencasts to show off new features or demonstrate how to do some things.  Sam and I have both prepared screencasts on volume rendering and getting started with developing, respectively.  Check them out below, and please feel free to leave comments and let us know what you think -- not just about the screencasts, but about what they demonstrate, and if you think any of the concepts or routines could be made easier.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Mon, 12 Sep 2011 11:53:00 -0700 Dataset Tracking with yt http://blog.yt-project.org/dataset-tracking-with-yt http://blog.yt-project.org/dataset-tracking-with-yt

In this post I'd like to discuss a bit of work in progress to highlight some exciting new features that we hope to have working in yt sometime soon.

On any machine that runs yt, there is a file created in the users home directory named ~/.yt/parameter_files.csv that yt uses internally to keep track of datasets it has seen. This is just a simple text file containing comma-separated entries with a few pieces of information about datasets, like their location on disk and the last date and time they were 'seen' by yt. To keep this file from exploding, it's kept at some maximum number of entries. But, clearly, text is not the ideal way to store this kind of information for anything over a few hundred entries. Recently Matt has been working on updating this system to use a SQLite database, which should have several advantages over the text file in terms of speed and disk usage.

This got me thinking about what could be done to extend this local listing of datasets into something more useful, globally. What if there was a way to view any and all datasets ever seen by yt in one convenient place? It could be searchable over a number of attributes, including creation date and when it was last seen by yt, and it would list which machine the dataset is stored on. Finally, this functionality should be transparent to the user once it is set up (with minimal effort) - the global listing of datasets should just be updated automatically in the background as part of the normal workflow.

Over a couple days last week I did a quick and dirty implementation of this using Amazon AWS SimpleDB and a simple web-cgi script I wrote in Python. The advantages of SimpleDB are that it is "in the cloud" (sheesh) and very inexpensive. In fact, for small databases with low usage levels, it is free. (As an aside, Amazon is very generous with academic grants, which could be used for this or other yt-related services.) The Python script is very simple and can be cloned off of BitBucket. The script can be run on any computer with a webserver and Python (which includes Macs and Linux machines), and I envision a website (perhaps mydb.yt-project.org, for example) being created where a user can login from anywhere to view their datasets easily.

The entire thing is not finished yet: the updates to SimpleDB are not automatic, nor have we settled on a final list of which attributes to store in the listing. However, in two days I was able to get enough working to show what I think are the key killer features of the system in a screencast which I've linked below. I should note that in the time since I made the screencast, I have made a few improvements. In particular, the numerical columns can now be sorted correctly.

I'm excited about the prospects for a simple system like this!

Permalink | Leave a comment  »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/heOzHGBSGnjGG Stephen Skory Stephen Stephen Skory Sat, 10 Sep 2011 09:39:52 -0700 Optimizing yt http://blog.yt-project.org/optimizing-yt http://blog.yt-project.org/optimizing-yt This last week, following the release of version 2.2 of yt, I spent a bit of time looking at speed improvements.  There were several places that the code was unacceptably slow:
  • 1D profiles (as noted in our method paper, even)
  • Ghost-zone generation
  • RAMSES grid data loading
The first of these was relatively easy to fix.  In the past, 1D profiles (unlike 2D profiles) were calculated using pure-python mechanisms; numpy was used for digitization, then inverse binning was conducted by the numpy "where" command, and these binnings were used to generate the overall histogram.  However, with 2D and 3D profiles, we used specialized C code written expressly for our purposes.  This last week I found myself waiting for profiles for too long, and I wrote a specialized C function that conducted binning in one-dimensions.  This sped up my profiling code by a factor of 3-4, depending on the specific field being profiled.

The second, ghost zone generation, was harder.  To generated a 'smoothed' grid, interpolation is performed cascading down from the root grid to the final grid, allowing for a buffer region.  This helps to avoid dangling nodes.  Ideally, filling ghost zones would be easier and require less interpolation; however, as we do not restrict the characteristics of the mesh in such a way as to ease this, we have to use the most general case.  I spent some time looking over the code, however, and realized that the most general method of interpolation was being used -- which allowed for interpolation from a regular grid onto arbitrary shapes.  After writing a specialized regular-grid to regular-grid interpolator (and ensuring consistency and identicality of results) I saw a speedup of a factor of about 2.5-3 in generating ghost zones; this has applications from volume rendering to finite differencing and so on.

Finally, in the past, RAMSES grids following regridding were allowed to cross domains (i.e., processor files.)  By rewriting the regridding process to only allow regrids to exist within a single domain, I was able to speed up the process of loading data, allowing it to preload data for things like projections, as well.  Next this will be used as a load balancer, and it will also ease the process of loading particles from disk.  I am optimistic that this will also enable faster, more specific read times to bring down peak memory usage.

Hopefully over the next few months more optimization can be conducted.  If you want to test out how long something takes, particularly if it's a long-running task, I recommend using "pyprof2html", which you can install with "pip install pyprof2html".  Then run a profiling code:

$ python2.7 -m cProfile -o my_slow_script.cprof my_slow_script.py
$ pyprof2html my_slow_script.cprof

This will create a directory called 'html', which has a nice presentation of where things are slow.  If you send the .cprof file to the mailing list, we can take a look, too, and see if there are some obvious places to speed things up.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Fri, 02 Sep 2011 07:56:00 -0700 yt-2.2 Release Announcement http://blog.yt-project.org/yt-22-release-announcement http://blog.yt-project.org/yt-22-release-announcement

Release Announcement

(Please feel encouraged to forward this message to any other interested parties.)


We are proud to announce the release of yt version 2.2.  This release includes several new features, bug fixes, and numerous improvements to the code base and documentation.  At the new yt homepage, http://yt-project.org/ , an installation script, a cookbook, documentation and a guide to getting involved can be found.  We are particularly proud of the new GUI, entitled “Reason,” which allows real-time exploration of datasets, and which can be used (locally or remotely over SSH) with no dependencies other than a web browser.  A basic demonstration of its usage can be found at: http://vimeo.com/groups/ytgallery/videos/28506477 .

yt is a community-developed analysis and visualization toolkit for astrophysical simulation data.  yt provides full support for Enzo, Orion, Nyx, and FLASH codes, with preliminary support for the RAMSES, ART, and Maestro codes.  It can be used to create many common types of data products such as:

* Slices
* Projections
* Profiles
* Arbitrary Data Selection
* Cosmological Analysis
* Halo finding
* Parallel AMR Volume Rendering
* Gravitationally Bound Objects Analysis

There are a few major additions since yt-2.1 (Released April 8, 2011), including:

* New web GUI “Reason,” designed for efficient remote usage over SSH tunnels
* Command-line submission to the yt Hub (http://hub.yt-project.org/)
* Absorption line spectrum generator for cosmological simulations
* Support for the Nyx code
* An order of magnitude speed improvement in the RAMSES support
* Experimental interoperability with ParaView
* Quad-tree projections, speeding up the process of projecting by up to an order of magnitude and providing better load balancing
* “mapserver” for in-browser, Google Maps-style slice and projection visualization
* Many bug fixes and performance improvements

With this release, we also unveil the yt Hub, an astrophysical simulation-specific location for sharing scripts, analysis and visualization tools, documents and repositories used to generated publications.  The yt Hub has been designed to allow programmatic access from the command line, and we encourage you to browse the current offerings and contribute your own.  The yt Hub can be found at http://hub.yt-project.org/ .

Documentation: http://yt-project.org/doc/
Installation:http://yt-project.org/doc/advanced/installing.html
Cookbook: http://yt-project.org/doc/cookbook/recipes.html
Get Involved:http://yt-project.org/doc/advanced/developing.html

If you can’t wait to get started, install with:

$ wget http://hg.yt-project.org/yt/raw/stable/doc/install_script.sh
$ bash install_script.sh

-->

Development has been sponsored by the NSF, DOE, and University funding.  We invite you to get involved with developing and using yt!

Please forward this announcement to interested parties.

Sincerely,

The yt development team

 

Permalink | Leave a comment  »

]]> Wed, 31 Aug 2011 03:58:50 -0700 This blog has moved -- but you don't have to! http://blog.yt-project.org/this-blog-has-moved-but-you-dont-have-to http://blog.yt-project.org/this-blog-has-moved-but-you-dont-have-to In keeping with the project rename we've moved this blog from it's old home at blog.enzotools.org to its new home at blog.yt-project.org.  But, we've put in a few redirects, and the RSS feed hasn't moved, so you shouldn't need to do anything different to get here.  We've also enabled anonymous commenting, so feel free to comment below.  (But, of course, letting us know who you are would certainly help with keeping in touch!)

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Wed, 24 Aug 2011 14:28:30 -0700 Introducing the yt Hub http://blog.yt-project.org/introducing-the-yt-hub http://blog.yt-project.org/introducing-the-yt-hub I am very excited to announce the yt Hub: http://hub.yt-project.org/.

The yt Hub is a gathering place for all things related to running, analyzing, and
presenting astrophysical simulations.  Once upon a time, we had the Enzotools
Barn, which was a place to share Enzo scripts unrelated to yt, including hosting
of those scripts.  The Barn was a minor success, but had a very limited scope
and a somewhat clunky user experience.

Now we have the yt Hub, where users can submit externally-hosted projects in
a variety of categories:
- yt scripts
- Analysis and visualization tools and utilities
- Simulation and data management
- Scripts for making figures from published papers
- News and announcements
- General astrophysical utilities

Users can create their own accounts and share anything they’ve created that might
be useful to others.  Currently, the preferred submission method is to create a
repository on bitbucket.org and then submit the link to that.  If you would like to host
the repository for your contribution somewhere else, that’s fine, too.  The forthcoming
2.2 release of yt will include a utility for automating the creation and uploading of a
bitbucket.org repository as well as submitting that repository to the Hub.

In addition, users may also subscribe to email announcements, in which they will
receive an email with any new submissions from the previous day.

Please, come check it out and contribute.  The site is brand new, so feel free to send
feedback if you encounter any oddities or think of improvements.  The primary goal of
yt has been to create a common analysis language for computational astrophysicists
coming from different simulation codes.  We hope that the yt Hub will serve as the
meeting place for the conversation.

Permalink | Leave a comment  »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/hdoupEkNdEbyO brittonsmith brittonsmith brittonsmith Mon, 22 Aug 2011 09:39:56 -0700 We're now the yt Project! http://blog.yt-project.org/were-now-the-yt-project http://blog.yt-project.org/were-now-the-yt-project This is just a brief blog entry -- other news is on the way soon! -- to let everyone know we've updated our domains.  In keeping with the change in focus of the yt project, expanding to work with multiple codes and take broader views of analysis and visualization, we have moved from enzotools.org to yt-project.org.  Some other exciting changes will be coming from this in the future, too!

For more info, see the mailing list announcement.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Mon, 08 Aug 2011 21:49:46 -0700 A movie of yt in paraview http://blog.yt-project.org/a-movie-of-yt-in-paraview http://blog.yt-project.org/a-movie-of-yt-in-paraview Jorge posted this video of using yt inside ParaView.  It demonstrated multi-res slices, projections, and even volume rendering with an adjustable transfer function.  I think it's pretty incredible.  Thanks very much to Jorge and Kitware for their hard work, and it'll be exciting to see where it goes from here!

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Wed, 29 Jun 2011 19:28:05 -0700 A mission statement and a new homepage! http://blog.yt-project.org/a-mission-statement-and-a-new-homepage http://blog.yt-project.org/a-mission-statement-and-a-new-homepage In case you haven't visited our front page lately, the yt homepage has been redesigned!  The old homepage was nice and functional, but it tried to cram too much into too small a place.  The new homepage focuses on the three main questions people have when they go to the yt page: "How do I get it?", "How do I use it?", and "How do I participate?"

The new page is focused on some concrete examples, some quantitative analysis, and features a heavy dosage of the yt community -- in fact, the redesign came about through a conversation in IRC, where a few of the core developers were speculating about how best to convey how to participate and how strong a community yt has.  The new homepage also features a mission statement.  (Discussion of the mission statement.)  It's designed to reflect the goals and ambitions -- not where the end of the road is, but which direction it is pointed.  To be honest, as the person who initiated that discussion, I was a bit surprised at how far reaching the discussion became.  The mission statement we converged on is reproduced here, but over time it might evolve and be refined.

The yt project aims to produce an integrated science environment for collaboratively asking and answering astrophysical questions. To do so, it will encompass the creation of initial conditions, the execution of simulations, and the detailed exploration and visualization of the resultant data. It will also provide a standard framework based on physical quantities interoperability between codes.

Development of yt is driven by a commitment to Open Science principles as manifested in participatory development, reproducibility, documented and approachable code, a friendly and helpful community of users and developers, and Free and Libre Open Source Software.

Feel free to stop by the IRC channel or the mailing lists if this piques your interest and you'd like to chat more.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt Mon, 13 Jun 2011 21:09:34 -0700 ParaView and yt http://blog.yt-project.org/paraview-and-yt http://blog.yt-project.org/paraview-and-yt

Thanks to some awesome work by developers at Kitware, yt can now be called from within ParaView!

This relies on a couple things, all of which are pretty exciting moving forward. The idea behind this is to leverage where the two codes have their own strengths and weaknesses, and identify places where they can work productively together. For instance, yt has been designed to provide astrophysical analysis; in this way, it can generate adaptive projections, spectral energy distributions and so on. By feeding data to yt (and not asking yt to read any on its own) these functions can be called from within ParaView, and images and results returned back to ParaView.

The strategy taken has, so far, been a light one. Almost all of the yt machinery has been left in place; what happens is a relatively simple process:

  • Load data into ParaView
  • Feed data into yt
    • Construct IO handlers that pass the contents of internal VTK structures to yt through a custom IOHandlerBase implementation.
    • Construct a hierarchy from the existing contents of the vtkHierarchicalBoxDataSet object, and its attendant child objects.
  • Execute analysis
  • Return processed data to paraview

We’re currently investigating how to streamline this process; currently, it utilizes a new frontend I wrote that was designed to stream data into yt from arbitrary locations. This can act as a proxy for datasets that are already loaded into yt, for datasets that are constructed ab initio in yt and for accepting data through in-memory transfer from ParaView! One of the fun applications of this will be using ParaView as a means for conducting [CoProcessing][http://www.kitware.com/blog/home/post/28] of data.

Thanks very much to Jorge, Berk, Charles and George at Kitware for all your hard work to make this happen. This is going to be a really fun way to explore new methods for analysis moving forward.

Not only is this exciting because it enables better cross-talk between yt and ParaView, but also because of the technology that is being developed on both sides. The stream handler in yt, in particular, is a great output from this: it can be used in the future to construct arbitrary datasets in memory, as well as to prototype new frontends for codebases. When yt gains the ability to write out datasets for various code types, this will be a valuable tool for constructing datasets from scratch.

For more information, see [this post][ http://public.kitware.com/pipermail/amr/2011-June/000027.html] on the AMR mailing list at Kitware, which contains an example script. While this functionality will be in the 2.2 release, it will be highlighted in the 2.3 release as we streamline and work out bugs in a proper ParaView frontend.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/492646/hs_05.jpg http://posterous.com/users/37lnBezefhIJ Matt powersoffour Matt