Google open-sources its protocol buffers

August 7th, 2008 | by admin |

Google says it has open-sourced its protocol buffers, the search giant’s system for encoding various types of data, in order to set the stage for a new wave of releases.

A FAQ page on Google’s site says “practically everyone inside Google uses protocol buffers. We have many other projects we would like to release as open source that use protocol buffers, so to do this, we needed to release protocol buffers first.”

Google says the XML (extensible markup language) isn’t very efficient enough for the company’s data-sharing needs. “When all of your servers and network links are running at full capacity, XML is a very expensive and rather cumbersome proposition,” said Google.

Kenton Varda, a member on Google’s software engineering team said “we use thousands of different data formats to represent networked messages between servers, index records in repositories, geospatial datasets and more.”

“Most of these data formats are well structured, not flat. This raises an important question: How do we encode them all?” added Varda.

“The answer is with protocol buffers. You simply define how you want your data to be structured once, then you can use specially generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of programming languages,” according to a documentation page.

The page continues with: “you can even update your data structure without breaking deployed programs that are compiled against the old format.”

However, XML still has some advantages for certain specific tasks, according to the documentation: “Protocol buffers wouldn’t be an efficient way to model a text-based document with a markup language such as HTML, since you cannot easily interleave structure within text. In addition, XML is human-readable and human-editable. For the most part, protocol buffers, at least in their native format, are not.”

On average, protocol buffers are three to ten times smaller and about twenty to one-hundred times faster than XML, according to Google.

The company has prepared a download page that contains protocol buffer compilers for Java, C++ and the Python language.