Flip Basics Programming Guide

Model Versioning

Over time, your model will change, and so its revision. Document then needs to be converted, and this chapter describes how to do so.

Converters

Your model folder should contain a conv folder. It will contains all your format version changes in a single place. Since Flip format conversion system is rather flexible, it is very easy to make document format changes, so you are very likely to make a lot of them.

You should name your conversion classes Converter#1_#2 where #1 would be the initial revision, according to your version source control naming. In the same fashion, #2 would be your destination revision.

Conversion graph

Just after reading a serialized document in BackEndMl or BackEndBinary to the BackEndIR format, you need to convert the latter format to the revision of the model currently used in your application.

BackEndMl or BackEndBinary are model revision agnostic, and so is BackEndIR, so that you don't need to maintain all versions of the model to be able to convert.

The general idea is to convert from one conversion to another the BackEndIR format until you've matched the revision available to use with your application.

Typically, this global conversion process looks like this :

DataProviderFile provider ("/path/to/file");
// tell the backend to automatically recognize the binary format
BackEndIR backend;
backend.register_backend <BackEndBinary> ();
bool ok_flag = backend.read (provider);
if (!ok_flag) return;   // a corruption occured
// now starts the conversion process, where every converters process
// the 'backend' in-place, and changes its version
if (backend.version == "myapp.beta.1")
{
   Converter_beta1_beta2 converter;
   converter.process (backend);
   // now backend.version == "myapp.beta.2"
}
if (backend.version == "myapp.beta.3")
{
   // beta.3 was a mistake, so we decide to rollback it to beta.2
   // for our loyal beta testers
   Converter_beta3_beta2 converter;
   converter.process (backend);
   // now backend.version == "myapp.beta.2"
}
if (backend.version == "myapp.beta.2")
{
   Converter_beta2_r100 converter;
   converter.process (backend);
   // now backend.version == "myapp.r100"
   // which is the current model revision
}
Document document (Model::use (), user_id, manufacturer_id, component_id);
// change the document with the content of the backend
document.read (backend);
// commit the changes
document.commit ();

Converter Overview

Now that we have seen the global conversion graph managment, let's look at a single converter anatomy.

BackEndIR contains a member version string which is the current revision of the model. Apart from that, it contains a member root which is the starting point of every conversions.

root itself is a BackEndIR::Type which is a generic way to represent a Type in the document. It can represent either an Object, but also a container type or a basic type like Float.

Type contains a type_name as well as a type_id to allows to know which kind of type we are manipulating.

Important: Usually you will want to process the backend in-place. The function converting the backend shall be completely deterministic, and in particular should not be dependent of time or any source of random.

The rationale behind it is to make sure that any one on the network to the same document can convert the document to a new version and have exactly the same result without requiring any kind of locking.

Adding a Member to a Class

Let's consider the following model :

class Note : public Object
{
public:
   String title;
   String body;
};
class Root : public Object
{
public:
   Collection <Note> notes;
};

And that we want to convert to this new model, where we added a member date

class Note : public Object
{
public:
   String title;
   String body;
   String date;
};
class Root : public Object
{
public:
   Collection <Note> notes;
};

Members in an Object in a BackEndIR Type are a list of pairs of a name and a Type, where the name is the one as declared in the Flip class declaration.

WARNING: Your converter shall be deterministic and shall not be dependent on current date or time, or any source of randomness.

Then the converter will need to add a member and initialize it to a value this way :

void  Converter_v100_v110::process (BackEndIR & backend)
{
   // !!! WARNING   we must not use the current date   WARNING !!!
   std::string default_date = "2017-01-12T13:26:07+00:00";
   // fetch 'notes' member
   auto & notes_member = backend.root.member ("notes");
   // member is a pair of name ("notes") and type (the collection itself)
   auto & notes = notes_member.second;
   for (auto & element : notes.collection)
   {
      // 'element' is a pair of key and type
      auto & note = element.second;
      note.object_add_member_string ("date", default_date);
   }
   // when the conversion is done, bump the current backend version
   backend.complete_conversion ("v110");
}

Renaming a Class

Let's consider the same starting previous model.

We now have two different classes for Notes which derive from it, so that a Note can either be a NoteText or a NoteImage. NoteText will have the same structure as the previous Note, but the class name change. Therefore we need to change every class that was named to Note to NoteText.

void  Converter_v100_v110::process (BackEndIR & backend)
{
   auto & notes = backend.root.member ("notes").second;
   for (auto & element : notes.collection)
   {
      auto & note = element.second;
      assert (note.get_class () == "Note");
      note.object_set_class ("NoteText");
   }
   backend.complete_conversion ("v110");
}

Renaming a Class's member

Let's consider the same starting previous model.

Let's suppose we now want to rename the member body to content.

Members in an Object in a BackEndIR Type are a list of pairs of a name and a Type, where the name is the one as declared in the Flip class declaration.

void  Converter_v100_v110::process (BackEndIR & backend)
{
   auto & notes = backend.root.member ("notes").second;
   for (auto & element : notes.collection)
   {
      auto & note = element.second;
      note.object_rename_member ("body", "content");
   }
   backend.complete_conversion ("v110");
}

Adding a New Object to a Collection

Let's consider the same starting previous model.

Let's suppose we want to add a new note for the user to discover a new feature of our software. For that we are going to synthesize a new object and add it to our container.

Elements in a Collection in a BackEndIR Type are a map of pairs of a key and a Type.

When manipulating a Collection in the model code, the key is just a random number. But because the converter processing function must be deterministic, we must use a pseudo source of randomness.

For this purpose, ConverterCollectionKeyGenerator is used. When adding an element to the collection, the generator is used to generate numbers. The generator will be called as many time as needed until it produces a key that is not already into the Collection.

Usually, one should use one generator for each converter, and seed it with a different number for each converter, mainly for performance reasons.

void  Converter_v100_v110::process (BackEndIR & backend)
{
   ConverterCollectionKeyGenerator generator (backend);
   auto & notes = backend.root.member ("notes").second;
   auto & note = notes.collection_add_object ("Note", generator).second;
   note.object_add_member_string ("title", "New Note Filtering feature");
   note.object_add_member_string ("body", "Filtering notes has never been easier!");
   backend.complete_conversion ("v110");
}

Adding a New Object to an Array

Let's consider the same starting previous model, with the only change that Notes are now ordered with an Array, so that Root now looks like this :

class Root : public Object
{
public:
   Array <Note> notes;
};

Let's suppose we want to add a new note for the user to discover a new feature of our software. For that we are going to synthesize a new object and add it to our container.

Elements in an Array in a BackEndIR Type are a map of pairs of a key and a Type.

When manipulating an Array in the model code, the key is a random floating number between two consecutive elements. But because the converter processing function must be deterministic, The key produced in the converter is just a fast computed center approximate of the two consecutive elements.

void  Converter_v100_v110::process (BackEndIR & backend)
{
   auto & notes = backend.root.member ("notes").second;
   auto & note = notes.array_add_object ("Note", notes.array.end ()).second;
   note.object_add_member_string ("title", "New Note Filtering feature");
   note.object_add_member_string ("body", "Filtering notes has never been easier!");
   backend.complete_conversion ("v110");
}

Adding a New Object to a Map

Let's consider the same previous model, where a Note has 3 members title, description and date.

class Note : public Object
{
public:
   String title;
   String description;
   String date;
};

This time we consider that we made a mistake into the original model, and that we want the concept of "dynamics object members".

Map can be used to do that. Map is very similar to the Object members description, except that "members" are ordered by names. Namely, Map is a map of pairs of a key and a Type, where the key is a name. The name itself can be a number or a string.

class Property : public Object
{
public:
   String value;
};
class Note : public Object
{
public:
   Map <Property> properties;
};

In the following example, we move all 3 members into a Map called properties, and make the conversion accordingly. Because a Map can only contain an Object (as opposed to a basic type such as String), we introduce a Flip class for that.

void  Converter_v100_v110::process (BackEndIR & backend)
{
   auto & notes = backend.root.member ("notes").second;
   for (auto & element : notes.collection)
   {
      auto & note = element.second;
      std::string title = note.member ("title").second;
      std::string description = note.member ("description").second;
      std::string date = note.member ("date").second;
      auto & properties = note.object_add_member_map ("properties").second;
      properties.map_add_object ("Property", "title")
         .object_add_member_string ("value", title);
      properties.map_add_object ("Property", "description")
         .object_add_member_string ("value", description);
      properties.map_add_object ("Property", "date")
         .object_add_member_string ("value", date);
      // remove old members
      note.object_remove_member ("title");
      note.object_remove_member ("description");
      note.object_remove_member ("date");
   }
   backend.complete_conversion ("v110");
}

Making a Modification Deep into the Data Model Tree

In the previous data model used, the deepness of the tree was small so that modifying objects was easy.

Now let's configer the following data model.

class Note : public Object
{
public:
   String title;
   String body;
};
class NoteBook : public Object
{
public:
   Collection <Note> notes;
};
class Root : public Object
{
public:
   Collection <NoteBook> notebooks;
};

Let's suppose we now want to rename the member body of Note to content.

We can use the walk function as a convenience to achieve that. The first parameter of the function is the BackEndIR Type to start from, and the second function to execute on every Type objects.

In the following example, we apply walk on the document root so that we process the whole document.

void  Converter_v100_v110::process (BackEndIR & backend)
{
   walk (backend.root, [](BackEndIR::Type & type){
      // skip every type that is not a "Note"
      if (type.get_class () != "Note") return;
      auto & member = type.member ("body");
      member.first = "content";
   });
   backend.complete_conversion ("v110");
}

Moving an Object within the Data Model Tree

Let's consider the same starting model. We now are considering to introduce the concept of NoteBooks and to move all Notes to a default NoteBook.

class Note : public Object
{
public:
   String title;
   String body;
};
class NoteBook : public Object
{
public:
   String title;
   Collection <Note> notes;
};
class Root : public Object
{
public:
   Collection <NoteBook> notebooks;
};

To do so, we will :

void  Converter_v100_v110::process (BackEndIR & backend)
{
   auto & notebooks = backend.root.object_add_member_collection ("notebooks").second;
   auto & notebook = notebooks.collection_add_object ("NoteBook").second;
   notebook.object_add_member_string ("title", "My First Notebook");
   auto & dst_notes = notebook.object_add_member_collection ("notes");
   auto & src_notes = backend.root.member ("notes").second;
   for (auto & element : src_notes.collection)
   {
      // add the element as a pair (key/value)
      dst_notes.collection_add (element);
   }
   backend.root.object_remove_member ("notes");
   backend.complete_conversion ("v110");
}

Handling Conversion Errors

Whenever an unexpected error is triggered, such as trying to access a non-existant member or trying to insert an element which key already exist in a container, the functions described above will throw using flip_THROW which in turn throws a std::runtime_error exception.

The user of the framework should also complies to this scheme for the unexpected errors that might be triggered while converting.

Since the whole conversion system is user side code, exception handling shall be provided by the user of the framework.

If the backend is corrupted but could be successfully translated, then one of the following errors can happen :

bool  load (Document & document, const char * path_0)
{
   DataProviderFile provider (path_0);
   BackEndIR backend;
   backend.register_backend <BackEndBinary> ();
   bool ok_flag = backend.read (provider);
   if (!ok_flag)
   {
      // the file format is corrupted or the file couldn't be read
      return false;
   }
   // call the conversion graph
   try
   {
      convert (backend);
   }
   catch (...)
   {
      // conversion failed, backend is corrupted
      return false;
   }
   // change the document with the content of the backend
   try
   {
      document.read (backend);
   }
   catch (...)
   {
      // backend is corrupted
      return false;
   }
   // commit the changes
   try
   {
      document.commit ();
   }
   catch (...)
   {
      // validation failed
      return false;
   }
   return true;
}

See Also

The present guide describes only the most important and basic scheme of document conversion.

More details are available in BackEndIR::Type reference documentation.