to YAML or not to YAML: that is a question...

2011-01-28T09:27:00.000-08:00

Recently Natan Sweet the author of the YamlBeans library left a comment on the wiki page in SnakeYAML stating that the library comparison is very poor. Indeed I am biased and I may not have precise and up-to-date information. We shall try to create a non-biased version of such a comparison table taking all the arguments into account. I hope together with Natan we can deliver a better version.

In the mean time I try to clarify some of my statements.

I clearly state where I see that YamlBeans does not follow the specification. When I tried YamlBeans some time ago it trimmed trailing spaces from scalars in literal style. May it has been fixed in the meantime. I may check it.
How can YamlBeans prove it works with Java 5 if it cannot be built with Java 5? How I can be sure that no Java 6 methods are called ?
Indeed I made a mistake stating that line number is not reported in case of an error. I overlooked it because it is sometimes shown as part of another exception deep in the stack trace. SnakeYAML together with line number shows some error context - a couple of lines of the YAML document to clearly identify the problem.
The quality of the YamlBeans parser is indeed higher then the one from JvYaml. YamlBeans is not mentioned next to JvYaml's bugs anymore. I am sorry.
By "Low level API" I understand the agreed way to get the low details of the underlying YAML document. It means that this API must expose Nodes and Events which part of the specification. Of course YamlBeans has some low-level details. Are they described anywhere ? Can they be used ? For instance, recently JRuby began to use SnakeYAML because its low-level API is fully compatible with the standard Ruby parser.
The fact that something can be done does not mean it is done. It took a reasonable amount of time and efforts to deploy SnakeYAML artifacts into a public repository. YamlBeans has not finished the task yet. When you use Maven then the source is automatically downloaded and added to your project in Eclipse by maven script. This is very convenient.

In the meantime I have created a few issues for YamlBeans to show my points. I think it is better discuss it one-by-one to avoid any misunderstanding in the future.

Once I have time I will try to prepare another version of the comparison table and before I publish it I will get in touch with Natan to avoid mistakes.

TODO for SnakeYAML 1.7

2010-03-23T02:58:00.000-07:00

I have recently received 3 enhancement requests for SnakeYAML:

1. Provide a simplified interface to process configuration files. No JavaBeans. No anchors and aliases. No implicit types. It can be used in projects like YAML Maven POM.

2. Use annotations.

3. Introduce namespaces for tags. This is requested to support JRuby where a YAML document may refer to either Ruby or Java instances.

All three require a substantial implementation.
What I miss is the feedback from those who may need these features. More real-world examples would greatly help to see the scope of the change.
Do you think they are useful ? Do you have an idea how the API may look like ? Feel free to provide your feedback.

Use the poll to indicate the features you want to go first.

Ruby and SnakeYAML

2009-12-08T05:29:00.000-08:00

Recently I have seen a post yaml in Java and Ruby where SnakeYAML was not chosen as the YAML parser.
The message is:
Because snakeyaml seems to have a syntactical problem with this YAML document we did no further investigations on snakeyaml.

But the real problem is that the document contains tags and the parser must be instructed how to process them.
Let us see what we can do.
The document:

--- !ruby/object:Test::Module::Object
sub1: !ruby/object:Test::Module::Sub1
  att1: []
  att2: 0
  att3: []
sub2: !ruby/object:Test::Module::Sub2
  att1: MyString
  att2:
  - entry1
  att3: 12345

We do not need a lot of code to parse the document:

Constructor con = new Constructor(TestObject.class);
con.addTypeDescription(new TypeDescription(TestObject.class, "!ruby/object:Test::Module::Object"));
con.addTypeDescription(new TypeDescription(Sub1.class, "!ruby/object:Test::Module::Sub1"));
con.addTypeDescription(new TypeDescription(Sub2.class, "!ruby/object:Test::Module::Sub2"));
Yaml yaml = new Yaml(new Loader(con));
return (TestObject) yaml.load(input);

Let's create a YAML document:

DumperOptions options = new DumperOptions();
options.setExplicitStart(true);
options.setExplicitRoot(Tags.MAP);
Yaml yaml2 = new Yaml(options);
String output = yaml2.dump(result);

This is the result:

---
sub1:
  att1: []
  att2: 0
  att3: []
sub2:
  att1: MyString
  att2: [entry1]
  att3: 12345

As you can see no tags are used. Normally SnakeYAML tries to avoid emitting tags as much as possible to simplify inter-language communication. But it seems that dynamically typed languages do not have any other means to define a class of an instance as to use a tag.
Let's emit the tags:

DumperOptions options = new DumperOptions();
options.setExplicitStart(true);
Representer repr = new Representer();
repr.addClassTag(TestObject.class, "!ruby/object:Test::Module::Object");
repr.addClassTag(Sub1.class, "!ruby/object:Test::Module::Sub1");
repr.addClassTag(Sub2.class, "!ruby/object:Test::Module::Sub2");
Yaml yaml2 = new Yaml(new Dumper(repr, options));
String output = yaml2.dump(result);

The document with tags:

--- !ruby/object:Test::Module::Object
sub1: !ruby/object:Test::Module::Sub1
  att1: []
  att2: 0
  att3: []
sub2: !ruby/object:Test::Module::Sub2
  att1: MyString
  att2: [entry1]
  att3: 12345

This document should be properly consumed by the Ruby parser. Unfortunately it requires to use the latest source from Mercurial. Current version (1.5) does not emit the tag for JavaBean properties even when the tag is explicitly defined. Either wait for the release after 1.5 or take the source !
SnakeYAML can work with YAML documents coming from Ruby parser.

Mission Accomplished.

Java libraries for YAML

2009-11-03T02:09:00.000-08:00

This is my overview of the state of different YAML libraries for Java. I do not claim it is complete (yes, I am biased) but it is my vision for today. Feel free to correct me.

JvYaml, v=0.2.1: the project is dead. No issue is fixed. The developer is now maintaining another project - JvYAMLb

JvYAMLb is a YAML processing library for Java, used by JRuby, v=0.2.5. I did not even evaluate the project because of a few dependencies on JRuby.

JYaml, v=1.3: last release is more then 2 year old. Many open bugs. I do not know whether it is maintained. Google puts this library to the top. Does it mean that it is the most wide spread ?

YamlBeans, v=0.93: since the parser and emitter are based on the JvYaml implementation it does not cover the whole 1.1 specification.
Implicit types are not supported (http://yaml.org/type/index.html)
Implementation does not follow the recommended way (http://yaml.org/spec/1.1/#id859333)
The document must be a JavaBean. Empty constructor is required. Immutable objects are not supported.

SnakeYAML, v=1.5. Unique features:

- Complete 1.1 YAML spec is implemented
- strong commitment to cross-language compatibility (as less global tags as possible)
- comprehensive documentation
- Maven 2 artifacts (including sources.jar and javadoc.jar)
- complete support for recursive objects (they can be parsed and emitted)
- 98% test coverage
- support for immutable instances
- collections in JavaBeans do not require global tags (other libraries need to configure additional runtime information which is required because of erasure)
- readable error reporting (with line numbers)

In fact I am not aware of any feature which is present in any library but not implemented in SnakeYAML. Nevertheless Google puts JYaml much higher then SnakeYAML and YamlBeans has much more downloads then SnakeYAML.
What is missing in SnakeYAML ? Why developers prefer to use another library ? What can be improved in SnakeYAML ?

Better support for type-safe collections

2009-10-20T08:01:00.000-07:00

Java 5 introduced a nice feature - type-safe collections (generic collections). Unfortunately due to erasure the actual type is removed by the compiler and there is no way to detect the type of objects in a collection.
That is why all the YAML parsers require additional configuration to parse a collection. SnakeYAML is using TypeDescription to detect the runtime class.
Fortunately if the collection is a JavaBean property there is a way to detect the runtime class. Thanks to Benjamin Bentmann for the idea.
Check the issue 25.


JavaBeanLoader beanLoader = new JavaBeanLoader(ListBean.class);
ListBean parsed = beanLoader.load(output);

These 2 lines parse ListBean with properly created Developer instances. Parsing JavaBeans with SnakeYAML has become even simpler!
At the moment only List is supported. Does anybody need to extend the support to Set and Map ?

Что же делать с рекурсивными объектами ?

2009-06-29T04:13:00.000-07:00

На данный момент SnakeYAML не умеет правильно работать с объектами, которые ссылаются на самих себя (напрямую или через другие объекты). Вот сама проблема. При этом имплементация сериализации (dumping) проще и это работает. Десериализация сложнее и имплементация потребует значительных изменений в коде при создании объектов.
Так как в настоящий момент у меня нет нужды работать с рекурсивными объектами, то никаких телодвижений в данном направлении не наблюдается. Если кому-нибудь захочется это починить, то все будут только рады.
В своё время я спросил у разработчика PyYAML, как это там сделано. Кирилл дал очень полный и исчертывающий ответ. (Спасибо!) К сожалению, в Jav'e не доступны те средства, которые используются для этого в PyYAML - generators. Поэтому импортировать реализацию в SnakeYAML не получится.
Для тех, кто не боится интересных задач привожу здесь ответ Кирилла:


Попробую объяснить так.  Нам дан ориентированный граф (YAML document) с выделенной вершиной - корнем документа.  У каждой вершины есть метка - tag.  Каждому тегу мы можем сопоставить конструктор - функцию, которая принимает в качестве параметра вершину графа и возвращает native object.  Задача состоит в том, чтобы преобразовать весь документ и вернуть native object соответствующий корню документа.

Как это сделать?  Самый простой случай - когда alias-ы вообще не используются.  Тогда граф представляет из себя дерево, корень дерева - это корень документа.  Алгоритм преобразования дерева в объект можно представить так:

чтобы преобразовать узел дерева:
   для sequence node: преобразовываем узлы входящие в
       последовательность (рекурсивно)
   для mapping node: преобразовываем узлы входящие в
       отображение (рекурсивно)
   преобразовываем сам узел и возвращаем его.

Не многим сложней и случай, когда alias-ы допускаются, но не создают циклов в графе.  В этом случае достаточно добавить в алгоритм проверку:

   если узел уже преобразован, вернуть соответствующий объект.

Проблема заключается в том, что делать, когда в документе есть циклы.
Самый простой пример:

--- &A [ *A ]

последовательность в которой один элемент - сама последовательность. Создать такую последовательность в Python-е можно так:

seq = []
seq.append(seq)

Тут важно заметить, что создать рекурсивную структуру за один шаг невозможно.  Здесь процесс преобразования разделен на два шага: создание объекта и заполнение объекта.  Для создания объекта не используются объекты соответствующие другим вершинам графа.  Это подсказывает идею: сперва создать объекты соответствующие всем вершинам документа а затем заполнить их.

Итак, мы разбиваем конструктор на две различные функции: создание и заполнение.  Для создания объекта не требуется знаний про другие вершины документа и соответствующие им объекты, для заполнения - требуются. Алгоритм выглядит так:

для каждой вершины документа:
   создать соответствующий объект.
для каждой вершины документа:
   заполнить соответствующий объект.

Это решает задачу в том случае, если мы способны разбить таким образом каждый конструктор.  К сожалению, это не всегда возможно: для некоторых типов создание и заполнение объекта всегда производится в один шаг. Например, в Python-е есть тип tuple - неизменяемый список, содержимое которого можно указать лишь при создании объекта:

t = (1, 2, 3)

Это делает невозможным создание tuple содержащей саму себя, т.е. объект,
соответствующий документу:

--- !!python/tuple &A [ *A ]

невозможно построить.  Тем не менее, это не значит, что tuple не может входить в сложные рекурсивные структуры.  Например документ:

-- !!python/tuple &A [ [ *A ] ]

можно преобразовать так:

l = []
t = (l,) # это конструктор tuple из одного элемента в Python-е
l.append(t),
return t

а документ:

--- &A [ !!python/tuple [ *A ] ]

так:

l = []
t = (l,)
l.append(t)
return l

Интересно, что в обоих случаях создание объектов происходит одинаково.

Итак, дело усложняется тем, что конструкторы бывают двух видов - такие, в которых создание объектов требует информации про другие вершины и такие - в которых создание и заполнение объекта можно разделить.  Что делать в этом случае?  Назовем вершину A зависимой от вершины B, если для создания объекта, соответствующего вершине A требуется знать объект, соответствующий вершине B.  Классифицировать зависимости в документе не сложно: если конструктор вершины разбивается на два шага: создание и заполнение, то вершина ни от кого не зависит.  Если же конструктор одношаговый, то вершина зависит от своих элементов.

Далее, отсортируем все вершины так, чтобы B находилась перед A когда A зависит от B.  Это делается с помощью так называемой "топологической сортировки" - варианта поиска в глубину в графе.  Если вершины отсортированы в нужном порядке, то к ним можно применять конструкторы, зная что объекты, которые требуются для преобразования вершины к этому времени уже созданы.  Итак, окончательный алгоритм выглядит так:

отсортировать вершины в порядке зависимостей, выдать ошибку если это невозможно.
для каждой вершины в указанном порядке:
   если вершина допускает многошаговый конструктор:
       вызвать первый шаг конструктора - создание объекта
   иначе:
       вызвать конструктор
для каждой вершины в указанном порядке:
   если вершина допускает многошаговый конструктор:
       вызвать второй шаг конструктора - заполнение объекта

Как видно, пример с взаимно рекурсивными списком и tuple вполне соответствует этому алгоритму.  В PyYAML описанные шаги алгоритма не выражены явно, однако по сути используется именно такой подход.

Буду безумно благодарен тому, кто не пожалеет времени перевести сие на английский, чтобы помочь желающим решить проблему.

Migration to Google Code

2009-06-08T09:01:00.000-07:00

Today SnakeYAML has successfully migrated from Assembla to Google Code.

Thanks to Google for providing Mercurial support. Now Google is used to host the whole project infrastructure:

Project home
source
wiki (documentation)
download

Google AppEngine

generated project reports
Maven 2 repository
InstantYAML (http://instantyaml.appspot.com/) application

The bigest problem was to convert documentation from one wiki markup to another. Fortunately they are not that different.

At Assembla (due to incredible amount of spam) I had to allow only registered users to create Trac tickets. Now anybody can report a bug or suggest a feature.

Performance, regular expressions and Ragel

2009-03-03T07:18:00.000-08:00

Recently I got a note that SnakeYAML is slower that JYaml. I have written a small stress test to load a document in a cycle. Indeed, it clearly showed that SnakeYAML performs bad with a big load.
Profiler could help to find the bottleneck - regular expressions. First, they are slow (I am afraid Just-in-Time compiler does not work here). Second, they scale badly, the bigger the load the slower they perform.
Regular expressions are used to support implicit types. Based on the data format SnakeYAML creates an appropriate Java class:
123 -> Integer
1.0 -> Float
false -> Boolean
2009-03-25 -> Date
abc -> String
and so on.
Of course we may take the JYaml's approach and drop regular expressions. Then all the scalars become Strings. It works but then developers must support all the (weird) formats like:
'1_000.5000_' -> 1000.5
23:59:59 -> int
Off -> false
~ -> null
etc.
Fortunately I came across Ragel. I gave it a try. It is cool. It creates an extremely fast implementation.
Regular expressions for implicit types are removed. First, we do not need to compile them when an instance of Yaml is created. Second, we do not match a long list of regular expressions against each and every scalar with an implicit type. As a result the stress test performs 2-3 times faster! This is impressive.
Of course if a single YAML document is loaded then the performance would not grow that much. This is because SnakeYAML has a number of static initializers for constants. Once the very first Yaml instance is created other instances are very chip to create. It should not have any significant influence on the performance.
This changes will be introduced soon in SnakeYAML 1.1.

InstantYAML is released

2009-02-03T04:28:00.000-08:00

InstantYAML is released today.
I hope it will help:

newbies to get easier the YAML concept
developers to come to a common denominator for what events, tokens and nodes are. Unfortunately they deviate significantly in the myriad of current YAML libraries.

PyYAML 3.05 is used to parse and emit the document. Once I have time I will update it to the latest version.

I must say Google AppEngine rocks. Development environment is very good and deployment is incredibly easy. It just took me a while to cope with the proxy...

2009-01-16T15:07:00.000-08:00

Finally I got some time to finish all the planned features of SnakeYAML 1.0. Everything I find useful in the existing YAML libraries is implemented.
Special thanks to

JvYaml. Even though nothing is left it gave a good starting point and I did not have to start from scratch and it was possible to overwrite the code component by component
YamlBeans for ideas how to implement JavaBeans support.

Version 1.0rc1 is released and if there would be no issues found the final 1.0 will be released soon.

The critics is welcome!!!

2008-11-11T03:57:00.000-08:00

I have created a space in Assembla to look closely at JvYaml and call it SnakeYAML (http://trac-hg.assembla.com/snakeyaml). The source migrated from CVS to Mercurial. Standard Maven folder structure is applied.
It is very convenient that JvYaml is a direct port of PyYAML. It is so easy to see the Python implementation and compare the deviations. It is even possible to debug two implementations in parallel on 2 computers ! (Synergy is dead useful).
Before the code is changed let us contribute tests. A lot of examples from the http://yaml.org/spec/1.1/ are created. Unfortunately a number of tests fails.
These are some deviation from original Python code:

Reader is dropped (in favor of java.io.Reader) and BOM is not respected. When stream is read the encoding must be known which is not always possible (and it is against the specification)
Scanner implementation is simplified. All the comments are removed.
Python implementation is not followed very closely. For instance a boolean in Python may be True, False and None. But Java implementation is using a primitive instead of the class Boolean and the third state is gone. It causes for example trimming the trailing spaces in the block scalars.
Python module is close to Java package. It helps separate code logically.
no tests are imported from PyYAML

Let us improve the implementation and try to follow the specification as close as possible.
This is what is done so far:

Java does not have multiple inheritance (which is very good!). The way how multiple inheritance is used in PyYAML is not very correct. Let us follow a reliable recommendation - "use composition over inheritance". Now Reader is an instance variable in Scanner.
Change the public interface and stay closer to PyYAML. Use Iterator instead of List. The java.io.InputStream is used and the encoding is recognized (and ignored) automatically
Rename classes with respect to "Python module" -> "Java package".
Define code formatter which can be imported to Eclipse
Go through ScannerImpl and try to stay as close as possible to PyYAML. A number of issues fixed. The size is almost doubled (~2000 lines), mostly because of the comments in the code.
some tests are imported from PyYAML

Because SnakeYAML provides some improvements over existing YAML libraries I can release the library.
Documentation is much worse then it should be. I will try to improve it later.
If somebody needs a reliable YAML parser for Java take SnakeYAML !
SnakeYAML 0.4 is born to this beautiful World...

2008-10-30T11:12:00.000-07:00

I have put YamlBeans 0.9.2 to my project. It works.
To see the internals better I have created a space in Assemla
http://trac-hg.assembla.com/yamlb

source code migrated from SVN to Mercurial
use Maven and apply the recommended folder structure

:( Mercurial is not perfect. When a resource is renamed it is not that simple to see the history in Eclipse.)

Because the project does not define a lot of tests I have started contributing tests to cover complete YAML 1.1. Once I have written most of examples from Chapter 2 (http://yaml.org/spec/1.1/) and most types (http://yaml.org/type/index.html) I realized that YamlBeans fails to cope with a lot of them.
The biggest problem is that most of types are not supported. Only strings, sequences and mappings work.
Read from the YamlBeans project page:

The YAML tokenizer, parser, and emitter are based on those from the JvYAML project. They have been heavily refactored, bugs fixed, etc. The rest of the JvYAML project was not used because of its complexity. YamlBeans strives for the simplest possible thing that works, with the goal being to make it easy to use the YAML data format with Java.

Unfortunately, an attempt to agree with the developer on what shall be supported did not lead to a common understanding. Apparently it should be not a big deal for Java projects.
But it creates an inconvenience if data is exchanged between Java and Python. For instance

map.put("text", "123")

will dump in YamlBeans
---
text: 123
...

instead of (like it is in JvYAML)
---
text: !!str 123
...

As a result PyYAML will assign to 'text' an integer which is not what I would expect.

Also YamlBeans will parse null, bool, float, int, binary as strings. Which is not always convenient.

The public interface was significantly changed and I need to learn 'yet another API' to work with YAML.

I think I shall give JvYAML a try.

2008-10-30T09:18:00.000-07:00

I need to deliver a RESTful service in Java. Restlet seems to be a good way to go. What should be the data format ? XML alone does not look very user friendly. Let me try to provide XML, YAML and JSON.
XML and JSON are well supported in Java. But the YAML support leaves much to be desired...
There are 5 open source implementations I could find:

yaml4j (http://yaml4j.sourceforge.net/)
JYaml (http://jyaml.sourceforge.net/index.html)
JvYAML (https://jvyaml.dev.java.net/)
YamlBeans (http://yamlbeans.sourceforge.net/)
JvYAMLb (http://code.google.com/p/jvyamlb/)

First three look dead. But JvYAML is exceptional because it is a port from PyYAML. In case of a question it is always possible to consult Python implementation.
YamlBeans is a new kid on the block and it is based on JvYAML. JvYAMLb has a dependency on JRuby which makes it impossible to use it in my projects.
So we are down to 2 libraries: JvYAML and YamlBeans.

JvYAML pros:

supports most of YAML 1.1
based on well supported project
the developer has a blog (http://ola-bini.blogspot.com/2006/09/announcing-jvyaml-021.html)

JvYAML cons:

all the issues stay open (https://jvyaml.dev.java.net/issues/buglist.cgi?target_milestone=milestone%201&issue_status=UNCONFIRMED&issue_status=NEW&issue_status=STARTED&issue_status=REOPENED)
anchors and aliases are dumped even when this is not needed, which reduces readability

YamlBeans prons:

very good support for JavaBeans
good output

YamlBeans cons:

reduced support for YAML 1.1 (most of types are not supported)
code significantly deviates from PyYAML

YamlBeans seems to be an easier way to go. Let's give it a try.