Tip: RVM + Pow

If you use RVM and Pow (http://pow.cx mac os x only), you will probably have to do this inside each of your ruby projects:

rvm env -- --env > .powenv

Bullet Points From WNY Ruby March Meetup

Here’s some notes from our march meetup (meetup page).

WNY Ruby March-a-palooza

`*.,*'^`*.,*'^`*.                         
^`*.,*'^`*.,*'^`*.,*'                     
'^`*.,*'^`*.,*'^`*.,*'^`*         ,---/V\  <  RUBY!! )
*'^`*.,*'^`*.,*'^`*.,*'^`*.,*    ~|__(o.o)
,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`  UU  UU 

Riak for the Ruby Brigade

by Andrew Thompson

Traits

• key value
• simple operations (get, put, delete)
• rest inspired API
• optional metadata
• mapreduce (like ruby map / inject)
• links (link to other keys, useful for associations)
• full-text search (via ria-search, bundled)
• secondary indexes
• distribution, clustered (recommended)
• fault tolerance (no master / slave), handles node failure
• all data written N times
• network partitions (ie switch dies)
• Highly available (ask any node to get data)
• live node inherits responsibility of dead node
• define strickness

Background

• built with erlang/OTP (open telecom platform)

Speaking REST

• HTTP default interface
• works well with reverse proxy caches, load balances, web servers, etc

Amazon dynomo inspired

• masterless
• consistant hashing
• anti-entropy: read-repair, hinted hand-off

Data Model

• referenced by its key
• grouped into buckets
• object composed of metadata and value
• metadata:
  • vector clock
  • Content-Type (text/html, text/plain, application/json, etc.)
  • Links

Consistent Hashing

• bucket/key combination is hashed to a 160-bit
• Ring is split into evenly sized partition, tries hard to prevent two partitions handled by nodes
• ring divided evenly as possible among nodes

Requests and magic numbers

• N - number of replicas (default: 2)
• R - read quorum (how many reads you want to succeed)
• W - write quorum (how many writes before return, accepted the write)
• DW - durable write (write actually happened, likely on disk)

every request will read every (N) replicas

Conflict resolution

• Conflicting writes, using a vector clock to determine which is write
• you can configure to allow multiple (could use application to deal conflict manually)

Node failure

1. node dies (had certain partitions)
2. partitions get redistributed to live nodes
3. requests re-routed, writes succeed, N-replicatas of writes created still
4. when node comes back up, revived node is updated

• read repair -> one replica didn’t get a write, that node gets pushed correct information

• can gracefully scale nodes

Clients

Official clients: erlang, java, ruby, python and php.

There are several un-official clients.

Storage engine

• Bitcask (default) - append-only, all keys stored in memory, disk for data
• Memory - all stored in memory
• LevelDB - by google, faster than Innostore
• Innostore - ‘Embedded InnoDB’, not supported by Oracle, GPL (not recommended)
• Multi - proxy to different backends depending on buckets name

Search and secondary indexes

• full-text search:

  • custom analyzer support
  • custom schema support
  • Lucene inspired query syntax (inspired, not exact and not fully)

• secondayr indexes

  • leveldb backend only
  • integer or string indexes
  • can be arbitrary
  • query on keys using secondary indexes, can do range (or exact) queries

Both of these let you feed the result to map/reduce.

Enterprise edition (commercial offering)

• materless multi-site replication (useful different regions), real-time / periodic (differential), can be bidirectional
• full customer support

Reasons not to use

• single server database
• simple relational model
• need to commit to using riak versus sql knowledge
• unsure of application data model, needs planning

Reasons to use

• Availability, assurance of uptime
• Scale predictibility
• latency predictibility
• willing

Riak success stories

• webapps: Yammer, Clipboard, GitHub (link shorteners)
• casual gaming
• medical records (Trifork + Denmark)
• smartphone applications (Voxer, Bump)
• caching (Posterous)
• session store

Voxer

pusth-to-talk phone application

• store both user and raw audio data in riak

Questions

Q: scaling, initial configure of ring size
A: hard to do, it is a pain point but is being worked on. append-only is
   cleaned out periodically

Q: learning knew way to structure data
A: ocassionally, relational databases fall over at certain scale levels.
   time-series (range queries), can do a rollup 

Intro to MongoDB

Ines, Data Engineer @ Engine Yard

Relational World

ACID

• sharding

NoSQL

CAP - Consistency, Availability, Partition Tolerance

• horizontal Scaling
• unstructured data
• large quantity of data

MongoDB

Document oriented DB (binary JSON), cschema-less, memory-mapped files (2GB) written in C++

Query interface

• js console

• will have aggregation framework (native, faster)

• sharding out-of-the-box

• map/reduce
• geospatial indexes
• backups
• imprort/export data
• single write lock (hinders heavy writes)

Structure

• single master DB
• roles: master and slave
• Replica sets:
  • 2-3 nodes basic install
  • Primary, Secondary & arbiter (Tie-breaker)
  • Automatic failover

Replica sets recommended (1.7+) over Master / slave setup

• replica sets: data
• config servers: sharding info, balancer
• mongos: connections to cluster, application slices

Oplog

• keeps track of replication
• capped collection: default 5% of total disk (EY uses 10%)
• size determines how much replication will be retained

The Journal

• write-ahead redo logs
• performance impact
• faster recover

Drivers

• mongomapper
• mongoid
• ruby mongo driver

Tips

• Use in 64-bits, 32-bits limits to ~2.5 gb addressable memory then kaputt
• Don’t run standalone, use replication
• use replica sets
• turn journaling on (default ~2.0.x)
• keep with current versions
• keep working set in memory
• use MMS for monitoring (mongo monitoring service)
• choose a good sharding key
• mind your config servers

WNY Ruby Meetup - March-a-palooza

Info

The meetup will be on Tuesday March 20th, 2012 at 7:00pm and be at our usual spot Caputis (2351 Sheridan Dr, Tonawanda, NY).

This meetup will be all about databases (relational and nosql) and star two out of towners. This will be a huge turnout (~40 people rvsp’d)

Speakers

Ines Sombra (@randommood)

from Engine Yard HQ in SF to come talk about MongoDB

Andrew Thompson (Vagabond)

from Basho to talk about Riak

Wayne Seguin (@wayneeseguin)

of Engine Yard to discuss PostgreSQL.

Ruby listen on Redis Queue, push to websocket channel

Here’s a fun bit of code.

Github Highlight: Annyong

Annyong starts a public static Web server in the current directory, allowing people in your local network to browse your files.

— remiprev, https://github.com/remiprev/annyong

A very simple and nice static file browser.

I liked it a lot, so i forked it and added audio tags for mp3|ogg files. A quick and dirty addition, but it works.

Get my fork of Annyong.

(Source: onethingwell.org)

Hidden in Rack-Contrib

Here’s some helpful Rack Middleware found in Rack-Contrib.

Rack::Evil

Force a certain response from anywhere. This is useful if you need to debug something from anywhere.

Rack::Profiler

“Uses ruby-prof to measure request time.”

Rack::Rackety (not in contrib)

Something I started to act as an API end-point to test connectors, somewhat like the other half of Hurl.

With Rackety, you can hit a url and have it respond with the contents of a file, such as sample JSON, XML, HTML, etc. You can also POST to it (right now it just returns the params, but will change to return sample response).

It’s an app right now since it’s not meant to really act as middleware, but I will reorganize it so it can.

Microapp: HDhours

I got tired of manually inputting my work schedule every week for work so I created this mini app that basically parses my work schedule and puts it into google calendar. It still needs some work, but it works for now.

HD Hours ( http://hdhours.heroku.com )

• Paste in your schedule and enter your initials along with google account information
• prevents duplication (by saving the last shift, probably should use a better method)