Joseph Hsu

RailsConf 2012 Live Coverage

Fri, 20 Apr 2012 13:57:00 -0400

Check Lanyard for notes / slides from the conference.

I will be Live Blogging RailsConf 2012 which will be from April 22 - 25 (including Ignite RailsConf the night before) as long as my phone has signal or I have wifi. Feel free to tweet at me ( @jhsu ).

Aso there’s a Twitter leaderboard for the hashtag #railsconf: http://railsconf.rubyonrails.io/

Tip: RVM + Pow

Thu, 19 Apr 2012 13:14:45 -0400

If you use RVM and Pow (http://pow.cx mac os x only), you will probably have to do this inside each of your ruby projects:

rvm env -- --env > .powenv

"Imagine if you are extended multiple job offers from different companies, and you are trying to..."

Fri, 23 Mar 2012 11:23:42 -0400

“Imagine if you are extended multiple job offers from different companies, and you are trying to decide which one you will accept. Imagine that the way you go about this is that you write down the things that matter to you from most to least and that you use 3-5 things at the top of that list to decide. Those are your decision drivers. What are they?”

- Ilya Bagrak “The one interview question I always ask”

Bullet Points From WNY Ruby March Meetup

Wed, 21 Mar 2012 13:21:00 -0400

Here’s some notes from our march meetup (meetup page).

WNY Ruby March-a-palooza

`*.,*'^`*.,*'^`*.                         
^`*.,*'^`*.,*'^`*.,*'                     
'^`*.,*'^`*.,*'^`*.,*'^`*         ,---/V\  <  RUBY!! )
*'^`*.,*'^`*.,*'^`*.,*'^`*.,*    ~|__(o.o)
,*'^`*.,*'^`*.,*'^`*.,*'^`*.,*'^`  UU  UU

Riak for the Ruby Brigade

by Andrew Thompson

Traits

key value
simple operations (get, put, delete)
rest inspired API
optional metadata
mapreduce (like ruby map / inject)
links (link to other keys, useful for associations)
full-text search (via ria-search, bundled)
secondary indexes
distribution, clustered (recommended)
fault tolerance (no master / slave), handles node failure
all data written N times
network partitions (ie switch dies)
Highly available (ask any node to get data)
live node inherits responsibility of dead node
define strickness

Background

built with erlang/OTP (open telecom platform)

Speaking REST

HTTP default interface
works well with reverse proxy caches, load balances, web servers, etc

Amazon dynomo inspired

masterless
consistant hashing
anti-entropy: read-repair, hinted hand-off

Data Model

referenced by its key
grouped into buckets
object composed of metadata and value
metadata:
- vector clock
- Content-Type (text/html, text/plain, application/json, etc.)
- Links

Consistent Hashing

bucket/key combination is hashed to a 160-bit
Ring is split into evenly sized partition, tries hard to prevent two partitions handled by nodes
ring divided evenly as possible among nodes

Requests and magic numbers

N - number of replicas (default: 2)
R - read quorum (how many reads you want to succeed)
W - write quorum (how many writes before return, accepted the write)
DW - durable write (write actually happened, likely on disk)

every request will read every (N) replicas

Conflict resolution

Conflicting writes, using a vector clock to determine which is write
you can configure to allow multiple (could use application to deal conflict manually)

Node failure

node dies (had certain partitions)
partitions get redistributed to live nodes
requests re-routed, writes succeed, N-replicatas of writes created still
when node comes back up, revived node is updated

read repair -> one replica didn’t get a write, that node gets pushed correct information
can gracefully scale nodes

Clients

Official clients: erlang, java, ruby, python and php.

There are several un-official clients.

Storage engine

Bitcask (default) - append-only, all keys stored in memory, disk for data
Memory - all stored in memory
LevelDB - by google, faster than Innostore
Innostore - ‘Embedded InnoDB’, not supported by Oracle, GPL (not recommended)
Multi - proxy to different backends depending on buckets name

Search and secondary indexes

full-text search:
- custom analyzer support
- custom schema support
- Lucene inspired query syntax (inspired, not exact and not fully)
secondayr indexes
- leveldb backend only
- integer or string indexes
- can be arbitrary
- query on keys using secondary indexes, can do range (or exact) queries

Both of these let you feed the result to map/reduce.

Enterprise edition (commercial offering)

materless multi-site replication (useful different regions), real-time / periodic (differential), can be bidirectional
full customer support

Reasons not to use

single server database
simple relational model
need to commit to using riak versus sql knowledge
unsure of application data model, needs planning

Reasons to use

Availability, assurance of uptime
Scale predictibility
latency predictibility
willing

Riak success stories

webapps: Yammer, Clipboard, GitHub (link shorteners)
casual gaming
medical records (Trifork + Denmark)
smartphone applications (Voxer, Bump)
caching (Posterous)
session store

Voxer

pusth-to-talk phone application

store both user and raw audio data in riak

Questions

Q: scaling, initial configure of ring size
A: hard to do, it is a pain point but is being worked on. append-only is
   cleaned out periodically

Q: learning knew way to structure data
A: ocassionally, relational databases fall over at certain scale levels.
   time-series (range queries), can do a rollup

Intro to MongoDB

Ines, Data Engineer @ Engine Yard

Relational World

ACID

sharding

NoSQL

CAP - Consistency, Availability, Partition Tolerance

horizontal Scaling
unstructured data
large quantity of data

MongoDB

Document oriented DB (binary JSON), cschema-less, memory-mapped files (2GB) written in C++

Query interface

js console
will have aggregation framework (native, faster)
sharding out-of-the-box
map/reduce
geospatial indexes
backups
imprort/export data
single write lock (hinders heavy writes)

Structure

single master DB
roles: master and slave
Replica sets:
- 2-3 nodes basic install
- Primary, Secondary & arbiter (Tie-breaker)
- Automatic failover

Replica sets recommended (1.7+) over Master / slave setup

replica sets: data
config servers: sharding info, balancer
mongos: connections to cluster, application slices

Oplog

keeps track of replication
capped collection: default 5% of total disk (EY uses 10%)
size determines how much replication will be retained

The Journal

write-ahead redo logs
performance impact
faster recover

Drivers

mongomapper
mongoid
ruby mongo driver

Tips

Use in 64-bits, 32-bits limits to ~2.5 gb addressable memory then kaputt
Don’t run standalone, use replication
use replica sets
turn journaling on (default ~2.0.x)
keep with current versions
keep working set in memory
use MMS for monitoring (mongo monitoring service)
choose a good sharding key
mind your config servers

Rails rendering partials

Fri, 16 Mar 2012 21:50:45 -0400

just an fyi

render "some/partial", collection: @things

is not the same as:

render :partial => "some/partial", collection: @things

thing won’t be available in the views in the first one.

WNY Ruby Meetup - March-a-palooza

Wed, 14 Mar 2012 15:00:00 -0400

Info

The meetup will be on Tuesday March 20th, 2012 at 7:00pm and be at our usual spot Caputis (2351 Sheridan Dr, Tonawanda, NY).

This meetup will be all about databases (relational and nosql) and star two out of towners. This will be a huge turnout (~40 people rvsp’d)

Speakers

Ines Sombra (@randommood)

from Engine Yard HQ in SF to come talk about MongoDB

Andrew Thompson (Vagabond)

from Basho to talk about Riak

Wayne Seguin (@wayneeseguin)

of Engine Yard to discuss PostgreSQL.

First attempt at d3.js

Wed, 22 Feb 2012 11:40:00 -0500

I just started looking into D3.js, a Data-Driving Documents framework for working with data and displaying using svg.

D3.js is powerful in what it can do (as seen from the examples) but by itself it isn’t specifically meant for graphing. Instead, it helps with positioning and calculations.

My first attempt was creating a line graph

Then, I created a simple sparkline graph along with a side scrolling animation:

http://bl.ocks.org/1885497

Ruby listen on Redis Queue, push to websocket channel

Sat, 11 Feb 2012 19:16:00 -0500

Here’s a fun bit of code.

Scratching Itches: Chat

Tue, 31 Jan 2012 12:00:05 -0500

Chat networks

I connect to multiple chat networks:

IRC
AIM
MSN
google talk
facebook chat (jabber)

Needs

I wanted to be able to connect to message a person or read messages from anywhere.

I also didn’t want to have to constantly connect just to check messages. I needed push notifications to my iPhone.

Solution

My solution allows for me to connect via any irc client, since irc clients are easily accessible on many devices.

Weechat (irc chat client in the terminal)
Bitlbee - allows me to connect to all my non-irc chat networks
ZNC (irc bouncer, keeps connections to bitlbee / freenode alive)
Prowl push notifications on iPhone 4s (there’s a Prowl module for ZNC that i use)
Colloquy on iPhone

This setup requires a server to run ZNC and Bitlbee.

This setup allows me to connect using any irc connection and let’s me disconnect with my client without losing connection to messaging. When I get messaged I get a push notification to my phone, then I can reconnect and reply however I choose, either via computer in weechat or through my phone in Colloquy.

#WNYRuby meetup, my "Using and Abusing Git Hooks" presentation

Tue, 10 Jan 2012 22:27:44 -0500

http://jhsu.github.com/githooks-presentation

Foo Ross Daaaaaaaaaaaaaaa thanks #reddit #secretsanta (Taken...

Fri, 16 Dec 2011 17:15:24 -0500

Foo Ross Daaaaaaaaaaaaaaa thanks #reddit #secretsanta (Taken with instagram)

Taken with instagram

Tue, 29 Nov 2011 11:30:35 -0500

Taken with instagram

Rails (2 & 3) Rake console

Sun, 24 Jul 2011 15:20:07 -0400

A little script (./script/taskrunner) to run a little rake console without having to start up the env everytime, works with environment variables too.

to get it set up, run this from your rails project directory:

curl -o ./script/taskrunner https://raw.github.com/gist/1072723/61b9eb6d458e643b3ea9e679d6501c388f3d3377/taskrunner.rb
chmod +x ./script/taskrunner
./script/taskrunner

Nyc at a greek place (Taken with picplz.)

Thu, 14 Jul 2011 15:13:59 -0400

Nyc at a greek place (Taken with picplz.)

(Taken with picplz.)

Wed, 13 Jul 2011 12:38:59 -0400

(Taken with picplz.)

Pafdle boat (Taken with picplz.)

Sat, 09 Jul 2011 16:03:34 -0400

Pafdle boat (Taken with picplz.)

Scratching Itches: Listening to Music

Wed, 22 Jun 2011 22:12:00 -0400

In this series I’ll be talking about little lifehacks I have found that work for me.

All my music is stored in one location, but I want to be able to listen to it from work, in the car or wherever I go.

I hardly ever plug my phone into my computer and I don’t want to have to manually sync my music over. I want to be able to listen to my latest music without syncing.

My criteria for a solution:

don’t want to have to sync anything
in browser
mobile accessible

And So I Found… Subsonic

iphone & android apps (which can cache songs onto the phone)
accessible in the browser
user authentication
last.fm scrobbling
playlist support
multi-user

The downside of running subsonic is that there is some setup. You need a dedicated machine to run the server and set up port forwarding through your router if you want to access it externally. Using no-ip.com helps so I don’t have to remember an IP address to access subsonic. Subsonic also has to transcode the audio to the bitrate of your choice so you can’t skip positions right away (unless the song is already cached on your device).

The music player also is flash (which isn’t so great on linux especially on a 64-bit OS).

Subsonic can also eat up bandwidth when streaming to something outside the network. I had to mitigate this using QoS on my DD-WRT router.

Long-term conclusion

I’ve been using this setup for almost a year and I’ve had no problems with it. I listen to my music on my phone, listen to music in the browser at work and home (no native music player needed).

weapon of #railsconf be afraid (Taken with picplz.)

Tue, 17 May 2011 23:46:21 -0400

weapon of #railsconf be afraid (Taken with picplz.)

(Taken with picplz.)

Sun, 15 May 2011 17:58:04 -0400

(Taken with picplz.)

flying blimp at work (Taken with picplz.)

Fri, 15 Apr 2011 10:49:08 -0400