Back to the main page Back to the documentation 1. Overview: FileUtils 2. Lua 3. Gadget 4. Ramses 5. Dictionary

Overview: FileUtils

Read and write some common filetypes. The Gadget/Ramses things can be improved.

Lua

LUA provide a good way of reading parameter files. We provide a simple LuaFileParser. Example parameterfile:

---------------------------
-- Example parameter file
---------------------------

-- An array of strings
AStringArray = {"Fratton", "752", "Jaduja"}

-- An array of doubles
ADoubleArray = {1.0, 2.0, 3.0}

-- A simple string
AString = "Hello"

-- A simple bool
ABool = true

-- A simple double
ADouble = 1.2

-- We can use simple formulas to set a variable
ANewDouble = ADouble * 2^10 * ADoubleArray[2]

...which can be read as follows:

  std::string filename = "input.lua";
  FML::FILEUTILS::LuaFileParser lfp(filename);

  //===============================================================
  // Read the parameters in the file. If optional and not found the value is set to the fiducial value
  // To throw and error if we don't find it use lfp.required (true) instead of optional (false)
  //===============================================================
  auto ABool        = lfp.read_bool                ("ABool",                   false, lfp.optional);
  auto ADouble      = lfp.read_double              ("ADouble",                   0.0, lfp.optional);
  auto ANewDouble   = lfp.read_double              ("ANewDouble",                0.0, lfp.optional);
  auto AString      = lfp.read_string              ("AString",      "Fiducial value", lfp.optional);
  auto AStringArray = lfp.read_string_array        ("AStringArray",               {}, lfp.optional);
  auto ADoubleArray = lfp.read_number_array<double>("ADoubleArray",               {}, lfp.optional);

Ascii

Read simple ascii files:

// Read a regular ascii files with nskip header lines and containing ncol collums
// nestimated_lines is the amount we allocate for originally. Reallocated if file is larger
// Not perfect for really large files due to all the allocations we have to do
DVector2D read_regular_ascii(
    std::string filename,
    int ncols,
    std::vector<int> cols_to_keep,
    int nskip,
    size_t nestimated_lines = 10000);

// As above, but include every line read with probabillity fraction_to_read
DVector2D read_regular_ascii_subsampled(
    std::string filename,
    int ncols,
    std::vector cols_to_keep,
    int nskip,
    size_t nestimated_lines = 10000,
    double fraction_to_read = 1.0,
    unsigned int randomSeed = 1234);

Gadget

This is just some very simple thing I wrote years ago to read and write Gadget files (files with position, velocity and ID). When working with MPI it has the option to just keep particles belonging to the local task (for cases where all tasks reads all the same files). Should also include a method where N tasks reads and distributes them to other tasks at some point (useful when reading very large files for which having all tasks read all files will be super slow). If you are just looking for a Gadget reader there is also a C++ library by S. Bird on github: GadgetReader. Anyway here is how to use it:

// Container to store it in
std::vector<Particle> part;

// Set up reader
GadgetReader g;
const std::string fileprefix = "../../TestData/gadget";
const bool verbose = true;
const double buffer_factor = 1.0;
const bool only_keep_part_in_domain = true;

// Read all gadget files and fill the data in part
// If only_keep_part_in_domain then with MPI we only keep particles belonging to local task
// If buffer_factor > 1 we allocate storage for more particles than needed
g.read_gadget(fileprefix, part, buffer_factor, only_keep_part_in_domain, verbose);

// More options:
// If we have elements stored in a different order or some fields missing
// then we can set it as follows before reading
std::vector<std::string> fields_in_file = {"POS", "VEL", "ID"};
g.set_fields_in_file(fields_in_file);

// If we just want to read a single Gadget file then we can do it as follows
g.read_gadget_single(filename, part, only_keep_part_in_domain, verbose);

There is also methods for writing a Gadget file to disk:

std::vector<Particle> part;
... create some particles ...

// Write a single GADGET file (per task)
FML::FILEUTILS::GADGET::GadgetWriter gw;
const double scale_factor = 1.0;
const double boxsize = 1000;
const double OmegaM = 0.3;
const double OmegaLambda = 0.7;
const double h = 0.7;
const int nfiles = FML::NTasks;
// Factor to convert user-units to positions in [0,box)
// (here we assume we have positions in [0,1) in part)
const double pos_norm = boxsize;    
// Factor to convert user-units to sqrt(a)dxdt in km/s
// (here we assume we have peculiar velocities in km/s in part)
const double vel_norm = 1.0 / std::sqrt(scale_factor); 
gw.write_gadget_single("test." + std::to_string(FML::ThisTask),
                       part.data(),
                       part.size(),
                       NumPartTotal,
                       nfiles,
                       scale_factor,
                       boxsize,
                       OmegaM,
                       OmegaLambda,
                       h,
                       pos_norm,
                       vel_norm);

Ramses

Ramses reader (reads only info and particle files). With MPI we have the option of only storing the particles that belongs to the local task. What we store depends on what you have in your particle class: if you only have a position then we only store positions, but if you have any of tag, mass, family, level, velocity etc. then we store that also.

// Container to store it in
// We check if the class has the methods set_id, set_family, set_tag, set_level
// set_mass, get_pos*, get_vel* and if they are present the data will be stored in the particle. 
// If not we ignore it
std::vector<Particle> part;

// Set up the Ramses reader
// output_number is X in output_0000X
// keep_only_particles_in_domain means each task only keeps particles in its own domain
// so after we are done the particles are distributed among tasks
std::string outfolder = "../../../TestData/";
const int output_number = 8;
const bool keep_only_particles_in_domain = true;
const bool verbose = true;
RamsesReader reader(outfolder, output_number, keep_only_particles_in_domain, verbose);

// Read all particles and store it in part
reader.read_ramses(part);

// More options:
// The fiducial file format is POS,VEL,MASS,ID,LEVEL,FAMILY,TAG, but
// if the format is different one can set it here and we can also set if we want
// to store the resulting data or not
std::vector<std::string> fileformat{"POS", "VEL", "MASS", "ID"};
reader.set_file_format(fileformat);

// To just read the ith file and store it in part
reader.read_particle_file(i, part);

After reading the positions will be in [0,1) and the velocities will be the dimensionless $\frac{a^2 \frac{dx}{dt}}{(H_0 B)}$ (standard RAMSES units).

Dictionary

A simple class to hold parameters of different types:

// Add some values
ParameterMap p;
p["AString"] = std::string("Hello");
p["ADouble"] = 1.0;
p["AnInt"]   = 5;
p["AVector"] = std::vector<double>{0.0, 1.0, 2.0};

// Fetch values from the map (if not found will throw an error)
auto AString = p.get<std::string>("AString");
auto ADouble = p.get<double>("ADouble");
auto AnInt   = p.get<int>("AnInt");
auto AVector = p.get< std::vector<double> >("AVector");

// Fetch values from the map (if not found it will use the fiducial value
auto AnotherInt = p.get<int>("AnotherInt", 10);

FML

A C++ template library for working with particles, grids and solving PDEs in parallel

Table of contents

Overview: FileUtils

Lua

Ascii

Gadget

Ramses

Dictionary