Link to the original paper

Previewing from http://www.goland.org/Scope-VLDB-final.pdf

Link to paper

HaLoop: Efficient Iterative Data Processing on Large Clusters – It extends MapReduce by adding programming support for iterative applications, and also improves their efficiency by making the task scheduler loop-aware and by adding various caching mechanisms

iMapReduce – It allows users to specify the iterative operations with map and reduce functions, while supporting the iterative processing automatically without the need of users’ involvement. More importantly, iMapReduce significantly improves the performance of iterative algorithms by (1) reducing the overhead of creating a new task in every iteration, (2) eliminating the shuffling of the static data in the shuffle stage of MapReduce, and (3) allowing asynchronous execution of each iteration, i.e., an iteration can start before all tasks of a previous iteration have finished.

Twister: A Runtime for Iterative MapReduce – It uses a publish/subscribe messaging infrastructure for communication and data transfers, and supports long running map/reduce tasks, which can be used in “configure once and use many times” approach. In addition it provides programming extensions to MapReduce with “broadcast” and “scatter” type data transfers. These improvements allow Twister to support iterative MapReduce computations highly efficiently compared to other MapReduce runtimes.

Peregrine – From the Sipnn3r folks.

iHadoop: Asynchronous Iterations for MapReduce – The iHadoop model schedules iterations asynchronously. It connects the output of one iteration to the next, allowing both to process their data concurrently. iHadoop’s task scheduler exploits inter-iteration data locality by scheduling tasks that exhibit a producer/consumer relation on the same physical machine allowing a fast local data transfer. For those iterative applications that require satisfying certain criteria before termination, iHadoop runs the check concurrently during the execution of the subsequent iteration to further reduce the application’s latency

Link to paper

An excellent source of information that explains the trade-offs between these two options is the paper – To BLOB or Not To BLOB: Large Object Storage in a Database or a Filesystem?.
Going by the discussions in the paper a trade off can be made based on the size of the objects stored. For small file sizes (in the order of few 100s of KBs) a database offers higher read throughputs. But as the file size approaches MBs the throughput of the file system increases faster than that of the database.

A key aspect that the paper brings out is the impact that disk fragmentation has on the performance of such a storage solution. One of the main reasons why access from a filesystem is faster is because of its ability to deal with disk fragmentation due to repeated updates. As the “storage age” (avg. number of times a file has been replaced) increases both read and write performance of a database degrades. Mainly because databases simply don’t have an automated way of dealing with the impact of fragmentation due to repeated updates. Defragmentation of a database requires explicit application logic to copy BLOBS to a new table (on SQL server).

So some of the requirements that a good BLOB database solution needs to address includes providing ability to defragment automatically. At the minimum it should tell you how fragmented a BLOB object is. On top of this it can offer other optimizations like in place defragmentation etc.

It feels like the idea of scalable “BLOB databases” in general (for the lack of a better term) is perhaps still nascent. Most BLOB management solutions (for audio, video or text) rely on distributed object stores like S3 or Ceph. Most of these don’t even offer metadata storage along with the data objects, leave alone offering specialized indexing and search capabilities. Its left to the applications that push data objects into these object stores to deal with the question of how to keep the database object metadata and the filesystem object data synchronized.
Attempts such as HSS from AOL and Haystack are in some ways a start. Although they are very far from becoming a specialized distributed database.

Link to the original paper

Key features
* Partitions data across many instances of Paxos state machines
* Automatically repartitions data across machines as the data volume increases or new servers are added. This feature is just awesome! Say good bye to manual sharding!!
* Scales up to trillions of database rows
* Supports general purpose transactions
* Provides a SQL based query language
* Configurable replication
* Externally consistent reads and writes
* Globally consistent reads across the database at a timestamp

Architecture
A single deployment of Spanner is referred to as a universe. In practice there is usually one universe per environment. Like for instance a development universe or production universe etc. It is further broken up into zones. Each zone is a unit of administration and represents a location which can house data replicas. Zones can be added to and removed from the universe while the system is running. The zone has a single zonemaster and many (100s-1000s) spanservers. Zonemasters assign data to spanservers. Spanservers serve data to the clients. Each zone also has a location proxy which helps clients locate spanservers that house the data. The universe also has an administrative console called the universe master that displays information about the status of all zones. The placement driver, as the name suggests is responsible for the transfer of data across zones. It also remains in touch with the spanservers periodically to fulfill their data movement needs.

A single spanserver controls about 100-1000 instances of a data structure called tablet. Each tablet is a bag/collection of mappings of the format

(key:string, timestamp -> string)

Because Spanner assigns timestamps to each piece of data the underlying data model resembles a multi-version database over a simple k-v store. The tablet’s state is persisted in a set of B tree like files and a write ahead log on a distributed file system called Colossus (the next gen of GFS).

Replication is provided by the spanservers by implementing Paxos on atop every tablet. The Paxos state machine is implemented so that the bag of mappings are consistently replicated. All writes to the tablet initiate the Paxos protocol while reads go to the nearest tablet directly. The collective of replicas constitute a Paxos group.
As is typical with Paxos one replica is elected the leader. On the leader replica the spanserver implements a lock table to implement concurrency control. Any operation that requires synchronization acquires a lock from the lock table. Leaders also run a transaction manager to support distributed transactions. The lock table and the transaction manager together provide transactionality. When a transaction involves persisting across two Paxos groups the group leaders coordinate to carry out a two phase commit.

Spanner layers a bucketing abstraction called the directory on top of this bag of mappings. A directory is a collection of keys that have a common prefix. Its also a unit of data placement. All the data within a directory have the same replication configuration. Spanner moves data between Paxos groups in a directory-wise manner. Directories can be moved for reasons such as improving data locality, balance load and resource usage etc. This movement happens dynamically while the system is still online. Normally a directory which is about 50 MB can be moved in about a few seconds. This move is not transactional as it can block other ongoing transactions.
A directory is also the smallest unit for which an application can specify geo-replication properties. You can control the number and types of replicas and their geographic placement.

For an Application
To applications Spanner exposes the abstraction of a semi-relational, schematized tables (with synchronous replication), SQL based query language and general purpose transactions.

Trade offs
The world of distributed systems has hitherto shunned the use of the two-phase commit protocol due to availability and performance issues. The designers of system make an interesting departure from this long held idea, and for some good reasons. Here is what they have to say

We believe it is better to have application programmers deal with performance problems due to overuse of transactions as bottlenecks arise, rather than always coding around the lack of transactions. Running two-phase commit over Paxos mitigates the availability problems.

When understood naively it may appear that the world of distributed systems has come a full circle in struggle for better scalability and availability!

Data Model
An application creates a database within a universe. Each database can hold a number of tables. Tables have rows and columns. In addition they also store versioned values for the data in these cells. Every table must have an ordered set one or more primary key columns. The primary key uniquely identifies each row. The whole table is a mapping between the primary key columns to the other non-primary key columns.

In a distributed database the partitioning scheme is the key to improved performance. While partitioning you want to keep data from related tables within the same unit of placement to the extent possible. In Spanner’s case this unit of placement is the directory. So what you want the client is to specify the group of tables that should be held within a single directory. You can do this using the INTERLEAVE declaration in the table creation step. See the paper for further details.

Transactions & Timestamp management
Spanner is the first system out there that assigns globally meaningful commit timestamps to distributed transactions. Spanner provides the guarantee that if transaction T-1 commits before transaction T-2 starts then T-1′s commit timestamp will be smaller that T-2′s. Spanner offers this guarantee at a global scale and is the first system to do so. This feature is enabled by the TrueTime API.

The paper goes on to describe the details of timestamp assignment in different transactional scenarios such as Read-Write transactions, Read Only transactions and Snapshot reads.

Real world experience
The buzz about Spanner has been around for a little while now. Experimental production trials with Spanner began since early 2011 as part of the rewrite of Google’s ad backend called F1. The F1 team chose Spanner for several reasons –
1) Removes the need to manually partition data
2) Synchronous replication and automatic failover
3) Strong transactional semantics

Link to the original paper

At a very high level it looks like to understand the working of HAIL we will have to look at the three distinct workflows the system is organized around namely –
1) The data/file upload pipeline
2) The indexing pipeline
3) The query pipeline

Every unit of information makes its journey through these three pipelines.

The Upload Pipeline

Here is where we begin. Typically one uploads the file to be analyzed into HDFS first and then executes a set of MR jobs on the Hadoop cluster. With HAIL you use a Hail client to upload the file. The client parses the contents of the file based on newlines and splits it up into blocks such that no row spans across blocks. Additionally the user can also specify a schema while uploading the file (much like in PIG). HAIL then converts all data blocks to a binary PAX representation. The good thing about this pipeline is that the whole transformation happens as the file is being written on HDFS. The blocks are not re-read from HDFS which will cause a lot of extra I/O. This significantly improves the write performance.

The client then contacts the Name node to get a list of data nodes. It then sends chunked PAX blocks to the first data node. When the data node receives the packet it immediately forwards the same packet to the next data node. It does this without flushing the contents of the packets and the checksum to the disk. This is the same with every data node that receives the packet. Contents are not flushed to the disk immediately.

On receiving a whole block worth of contents the each data node sorts the contents and creates indexes based on the specification of sort order and indexes by the user. Each data node sorts the data in a different order. All the index metadata, the sorted data and checksums etc form what is known as a HAIL block.

Within HAIL its vital that the MR jobs run such that they are able to leverage the indexing thats happened on the data nodes. So the tasks have to be scheduled on the data node that has the most suitable index. In order to enable this the sort/index metadata has to be stored at the name node level. An instance of HAILBlockReplicaInfo contains detailed information about the types of available indexes for a replica, i.e. indexing key, index type, size, start offsets etc.

The Indexing Pipeline

The basic purpose of the indices is to get to the relevant blocks by scanning the index first. After experimenting with a few different types of indexes they seem to have concluded on using a sparse clustered B+ tree based index. The column that needs to be indexed is first sorted in memory and then the index tree is written to the disk on to a single directory. Note that the index is not a multi-level index. The paper gives some back of the envelope calculations for this choice.

The Query Pipeline

Much of the MR job continues to be written just as before but with some interfaces changed. Firstly the InputFormat implementation used is HailInputFormat. The other nicety of this framework is the typical task of filtering the records which is carried out within the map function can be delegated to HAIL. You can annotate the map function with @HailQuery annotation where you may declaratively specify the projected attributes and the filtering condition.


@HailQuery(filter="@3 between(1999-01-01,2000-01-01)", projection={@1})
void map(Text key, HailRecord v) { ... }

The HailRecordReader which collaborates with HailInputFormat is the component that applies the predicate to filter out the qualifying records. Lastly the value passed to the map function is a HailRecord object.

Summing it up
HAIL tries to support per-replica indexes in an efficient way and without significant changes in the standard execution pipeline. It tries to achieve much of this by providing alternate implementations of the InputFormat and RecordReader interfaces along with a custom splitting policy. HAIL improves upload and query times without impacting the failover properties of Hadoop and minimal change to the map reduce programming interface.

Link to the original paper

Data is stored on dedicated storage nodes, called tractservers (comparable to HFDS name nodes). The tractserver is a network front-end to a single disk; machines with multiple disks have one tractserver running per disk.
User code does not run on tractservers; applications can only retrieve data from or write data to tractservers over the network. In FDS, there is no such thing as a local file.

Contrary to the idea of moving compute to the storage nodes this system works by always sending data over the network. It manages the cost of data transport by
a) Giving each storage node network bandwidth that matches its storage bandwidth
b) Interconnecting storage nodes and compute nodes using a full bisection bandwidth network

This combination produces an uncongested path from remote disks to CPUs, giving the system an aggregate I/O bandwidth essentially equivalent to a system such as MapReduce that uses local storage. FDS also supports data replication for failure recovery.

In FDS, data is logically stored in blobs. A blob is a byte sequence named with a 128-bit GUID. Blobs can be any length, limited in size only by the system’s storage capacity. Reads from and writes to a blob are done in units called tracts. Each tract within a blob is numbered sequentially starting from 0. Tracts in FDS are about 8MB. The FDS API defines simple CRUD operations to interact with a Blob. All calls in the API are non-blocking. Consequently the API also takes in callback function that is invoked after the operation completes.

By spreading a blob’s tracts over many tractservers and issuing many requests in parallel, many tractservers can begin reading data off disk and transferring it back to a processing node in parallel. Deep read-aheads enable a tract to be read off disk into the tractserver’s cache while the previous one is being transferred over the network.

Does it have a SPOF?

A single central metadata server that should be consulted to learn about where the data is placed is a common design pattern in distributed storage systems. Writers contact the metadata server to find out where to write a new block; the metadata server picks a data server, durably stores that decision and returns it to the writer. Readers contact the metadata server to find out which servers store the blocks to be read. This approach turns the metadata server into a SPOF as it is always on the critical path for all reads and writes.

This system too has a metadata server, except that its role during normal operations is simple and limited: collect a list of the system’s active tractservers and distribute information about them to clients. This list known as the tract locator table (TLT), is first retrieved from the metadata server when a client starts. The metadata server stores only metadata about the hardware configuration, not about files.

When it wants to read or write a tract, it first computes a tract locator. The simplest tract locator is the sum of the 128-bit blob GUID to be read and the 64-bit tract number to be read, modulo the number of entries in the TLT. Indexing the tract locator into the TLT yields the tractserver to which that tract read or write should be issued. The TLT changes only in response to cluster reconfiguration and not individual CRUD operations. It can thus be cached by clients for a long time.

Since the tractservers remember their position in the table, the metadata server stores no durable state; in case of a metadata server failure, the TLT is reconstructed by contacting each tractserver. The TLT is never modified due to reads and writes.
Also the TLT contains random permutations of the list of tractservers. This increases the chances of sequential reads and writes by independent clients utilizing all tractservers uniformly. The TLTs independent permutations prevent clients from organizing into synchronized convoys.

Section 2.2.1 very splendidly describes the optimizations and trade-offs in the design of the metadata server. A must read!

Per-blob metadata, such as blob length and permissions, are stored in a special tract (“tract -1”) of each blob. Clients find a blob’s metadata using the same method for finding data, using the TLT. Thus per blob metadata management is as distributed as blob
data storage.

The rest of the paper describes the execution of the sort algorithm in great detail.