Father’s Day

By stephen | June 15, 2008

If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!

It is time for a great blog post, and, being that it is father’s day, I have the perfect topic. I wanted to write this post to thank my dad, Jeff Barr, for being a great father. Being his son and part of the family is a blast, and there are many reasons why. For example, most people my age joke about the technical incompetence of their parents, or their lace of ability to send an email or text message. I have the opposite problem. I struggle to stay maintain 25% of the competency that my dad has regarding technology, and life in general. My dad is the kind of person that can talk to an expert in ANY discipline - computer science, astronomy, finance, economics, manufacturing…anything except sports…., and meaningfully participate as an equal.

Additionally, my dad is always doing something interesting. I have never seen him watch TV by himself. If a family member is watching, he’ll sit down to keep them company for a while, but he never vegetates (his term for TV watching and other forms of passive entertainment).  This attitude has given everyone in the family the work ethic needed to accomplish great things. If you think you can keep up with him at a conference or business trip, think again. There are few that can.

I am at the point in my life where the professional interests of myself and my dad are thoroughly intertwined and it is fun to have him involved in my academic life. Most recently, he and another guy, Yair,  both from Amazon Web Services, were able to attend a meeting with myself and some finance professors at the UW to discuss Amazon’s role in the research we have been doing. It would have been a great meeting even if the AWS people were strangers, but having one of them be my dad made it that much better.

Dad, Happy Father’s Day!

Topics: General | No Comments »

distributed matlab (part 1 of 3) - compiling

By stephen | May 7, 2008

This is part one of a three part series on writing distributed applications with matlab. My particular interest is the analysis of financial data, but a knowledge of developing distributed matlab applications should be of use to people in any field where processing a large amount of information is critical, and matlab is your tool of choice.

There are several different ways to go about distributed computing with Matlab. Given your matlab m file, foo.m, you could

I prefer the third option, because I can then run my app on a Hadoop cluster running on EC2. Yeah, it’s awesome.

The first step in this process can be a bit cumbersome. Here are the issues that I came accross.

Suppose you had some m file, foo.m:

function g = foo()
% the “Squibonacci” function from the compiler documentation
n = 1000;
g = zeros(1,n)
g(1) = 1;
g(2) = 1;
for i=3:n
g(i) = sqrt(g(i-1)) + g(i-2);
end

The first step is to setup your Matlab compiler by running the Matlab command: mbuild -setup
Be sure to consult your documentation if something strange appears. Importantly, on Unix, this creates a file in your home directory called mbuildopts.sh. This controls to parameters that Matlab can use to compile and link the C files that you generate from the M-files. The file is located in ~/.matlab/R14/mbuildopts.sh.

At this point, try compiling foo.m.
>> mcc -m foo.m

If you are successful, this should have generated some files in the same directory, namely  foo_main.c, foo.ctf, foo_mcr, foo_mcc_component_data.c. The first time I did this, I ran into an error.

Warning: Duplicate directory name: /usr/local/matlabR14/toolbox/local.
/usr/bin/ld: warning: libstdc++.so.5, needed by /usr/local/matlabR14/bin/glnx86/libmwmclmcrrt.so, may conflict with libstdc++.so.6

The problem was that my default g++ compiler is version 3.4.6, and I needed to be using 3.2.3, such that I would link to libstdc++.so.5, rather than so.6. Luckily, my machine also had a copy of g++ 3.2.3 installed, named g++32. *NOTE - you can check your version of g++ by running g++ - version* To point Matlab at the correct g++, I modified mbuildopts.sh, replacing gcc with gcc32 in the appropriate section. This fixed the problem. (Note, you may need to dig up and install 3.2.3 if it is not on your system. There is some care needed to have multiple versions of gcc installed on the same computer. Make sure to look into this.)

After running mcc, it is time to run mbuild. Issue the command:

>> mbuild foo_main.c foo_mcc_component_data.c -g -output foo.x

This will compile a standalone executable called foo.x. Try to run it by going to a shell and doing ./foo.x.  This should produce the same output that would have been produced by running foo.m within Matlab. However, this is its own standalone application and can be run on systems without Matlab installed.  And, importantly for Hadoop streaming, you can control input and output to foo.x by redirecting STDIN and STDOUT.

Another possible pitfall is that, when running foo.x, you get the following error:

bash-3.00$ ./foo.x
./foo.x: error while loading shared libraries: libmwmclmcrrt.so.7.2: cannot open shared object file: No such file or directory

In this case, there is a problem with your  LD_LIBRARY_PATH environment variable. You can fix this by editing your ~/.bashrc file and adding these lines (these paths are correct for my box. They should be somewhat similar for other systems, at least those running Red Hat):

export MATLAB=/usr/local/matlabR14
export LD_LIBRARY_PATH=$MATLAB/bin/glnx86
export LD_LIBRARY_PATH=$MATLAB/extern/lib/glnx86:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/matlabR14/sys/os/glnx86:$LD_LIBRARY_PATH

At this point, you should be able to compile Matlab applications and be able to run them on the command line. Check back for part 2 of 3, packaging and distributing standalone matlab applications for running on Hadoop notes.

-steve

Topics: Economists' Toolbox | 1 Comment »

economics of suburbia and the transportation problem

By stephen | March 31, 2008

 00108.jpg

These are tickets from my recent trip to the East Coast. One thing I enjoyed there was the public transportation system. I really enjoyed being able to go anywhere in Manhattan with ease and for a low cost (a subway ride is $2, an all day pass is $7.50). From New York, I was able to go to DC for a few days to visit my relatives. This was also easy and inexpensive ($70 - $120 for an Amtrak ticket). Amtrak trains are great. You get on the train just a few minutes before it boards, and you get to read or work on your laptop while you go quickly and reliably to your destination. There are 120V AC power outlets available, so battery life isn’t an issue. If I had mobile broadband, I would even have been able to do some of my internet-based work. The point is, efficient and reliable public transportation is a great amenity of well developed cities and states.

It will also be a solution to the mess we have created for ourselves in terms of suburban sprawl. This mess has roots in economics. For a long time, it was relatively inexpensive for each person to have their own acre of land and large house. Government road development projects, combined with a postal service with universal service obligations (the guarantee that each person has the right to have mail delivered to them) made it easy for suburbia to develop.

The rising costs of energy are changing the costs of the suburban lifestyle to which Americans have grown accustomed.  This NPR broadcast/article talks about how in Atlanta, the average commute is a 66 mile round trip. That is crazy. As energy costs rise, the opportunity costs of living in the suburbs will increase, and people will begin to re-urbanize in order to save money. Also, shared parks, water heaters, air conditioning units, and transportation systems allow for the economies of scale to function, reducing energy costs.

From one point of view, it is good that energy is expensive and the expectation is that prices will continue to rise. This is because it will force innovation that will ultimately be for the greater good.  An example would be hybrid cars. As energy costs rise, it becomes more cost effective to research these technologies. The propagation of this technology has allowed for more energy efficiency, and as a side benefit, cleaner air. If gas was still $1/gal, we would still be driving around SUV’s and pickup trucks, just for the fun of it.

In a few years, I imagine that it will be too expensive for most people to live in such a way that they drive a 6000 lb vehicle 66 miles per day. And even if they could afford it, it seems wasteful. I look forward to seeing the innovations that will come as we are forced to live under these changing economic conditions. Wouldn’t it be nice to have cities with nice parks, with people using either mass transit or electric vehicles** to get to and from work. That would give the urban amenities (nice communities, stores and restaurants in walking distance, lots of people to talk to) without the pollution. That would be nice.

Take care!

~~~~

**the idea that electric cars help the environment is based on the assumption that the energy is coming from a clean source. If we use coal or oil burning power plants to generate the electricity to run the cars, we haven’t really solved the problem. Personally, I think that the answer, at least in the short and medium term, is nuclear power. It is reliable, clean, extremely efficient, and it is a well developed technology. Unfortunately, as Howard Dean taught us, one slip up can tarnish an otherwise decent track-record. I think it is time that, as a culture, we took another look at nuclear power.

Topics: General | 2 Comments »

What will happen today?

By stephen | March 26, 2008

3 months of Bear Sterns, via Google finance

bsc.png

It is up from $2 a share, which is good.

Topics: General, News | 1 Comment »

field trip

By stephen | March 24, 2008

I decided to go have right to the source concerning the latest rate cut.

the Fed - thumb

I also had to check out the stock exchange:

NYSE

A full trip report is coming soon.

Topics: Uncategorized | 3 Comments »

I’ll be watching you

By stephen | March 19, 2008

On the day after a 75 basis point cut by the Fed, I thought that showing this video would be appropriate.

Nice.

Topics: Uncategorized | No Comments »

How to educate your kids

By stephen | March 18, 2008

my 4th grade class

My 4th grade class. I am in the top row, with the red sweater.

There has been some discussion about education lately. In the news, a California District Court ruled that it is illegal for parents who are not certified teachers to homeschool their children. San Diego Tribune Article here

There are several different issues at stake here.

  • Is it the parent’s responsibility to educate the kids?
  • Is it the government’s responsibility to educate the kids?
  • Are certified teachers better than parents at teaching?

I don’t have a strong opinion on homeschooling. I think that for very special cases, it can be a good option, but otherwise, a larger institution such as a public or private school, is better. My strong opinion relates to the first bullet point - Is it the parent’s responsibility to educate the kids? To that, I have a very strong opinion: YES! How they implement that responsibility is up to them. Here are their options:

  • Move to an area with good local public schools. Be involved with the PTA. Pay taxes. Donate money.
  • Find a good private school. Be involved with the PTA. Pay tuition. And more tuition.
  • Homeschool (unless you are unlicensed and live in California)

But, all of these options are not important if the parents aren’t actively involved in their children’s education. School, no matter where it comes from, should only be a part of an education. It is the parent’s responsibility to instill a sense of curiosity and work ethic in their kids, so that they can have academic pursuits outside of the classroom.

In my personal life, I was always interested in computers. I spent quite a bit of time teaching myself programming - beginning with QBasic, Visual Basic, C and C++, Java, etc., and I could always go to my dad for advice. I would often ask for interesting programming problems to work on, for which my dad has an endless supply. This was an invaluable part of my education that was totally unrelated to anything that I was doing in school. I remember a particular evening when my friend Blake was over, and we asked my dad for an interesting problem to solve. He told us to write a program to multiply matrices together. It is 10 years later and I still use matrix multiplication every day.

My “tech” classes in middle school were incredibly weak by comparison - rudimentary review of HTML and a little image editing…stuff I had taught myself as a little kid. In fact, when I got to these classes, I immediately proved that I knew anything that was going to be covered that quarter, and was told to go “fix the scanner in that room over there”. The problem was a loose power cord.

To the point, if a kid’s education comes solely from school, they are in a lot of trouble. There is an old adage, “Never let school get in the way of an education.” If your kid is learning something in school, at home it should be taken to at least the next level. There is no reason why a parent couldn’t take their own initiative to teach more than the public school curriculum. Schools are designed to cater to a classroom full of kids, each with different abilities, work ethics, attention spans, and interests. It would be impossible to take each kid to their full potential without hurting everyone else. So, it is the parent’s responsibility to fill that gap between what the public school is doing and the potential of the child. Schools are important for providing an opportunity to develop social skills, group problem solving skills, the ability to understand and react to authority figures, and so forth. Additionally, they provide a good place pick up some good bacteria and viruses in order to develop a strong immune system, which is very important. However, formal schooling should not be the limit of an elementary education. If there isn’t additional stimulus, then there will be wasted potential.

I remember in 4th grade, there was a small group (4-6) of kids gathered together who were particularly good at math. I was included in this group. On one occasion, my dad got to come and talk to this group and try to teach us something interesting. He taught about different number bases, particularly binary, octal, and hexadecimal. These are concepts that most 4th graders would have had no exposure to in the normal curriculum. If it wasn’t for people who realized that our little 4th grade minds had room for more, then that immature gray matter would have been underutilized, and what a shame that would have been. The human mind has limitless potential, so it makes sense to try to explore that potential and push its limits. If an 8th grader is ready to learn calculus, then there no reason to wait till the senior year of high school. It will only be to their advantage if they have a solid foundation in calculus earlier in life.

Reading is another area where so much can be done above the what is assigned in schools. I am grateful that my parents taught me, through example, to love reading. They never worried if anything was “too advanced”. As a kid, I read everything I could find by Arthur C. Clarke, Ray Bradbury, and Isaac Asimov. I loved (and still love) 2001, A Space Odyssey, Farenheit 451, I, Robot, and everything that those great authors produced. Sometime in 5th/6th grade, I found a collection of Stephen King books and read all of them - IT, Cujo, Christine, Carrie, Needful Things, the Shining. That was a frightening year; Leland Gaunt was a scary guy. The point is, it is important to love reading. I am glad my parents taught me to do that. I intend to implement these ideas and policies in my future family.

As people research, ponder, and make new discoveries, they add to the general body of knowledge and advance the human race. Knowing how to teach yourself, and to be able to find things to do that expand your mind are essential characteristics that must be developed in children as a part of their education.

Topics: General | 1 Comment »

So, the Fed is in the mortage business?

By stephen | March 12, 2008

For an introduction, I want to start with this article:

Britnew

It appears that Ms. Spears is having some financial problems. Here are some paragraphs from the article that may be of interest:

What little light has been shed on Ms. Spears’s finances came last May, when part of an income statement surfaced in her divorce case. It put her average monthly earnings at more than $730,000, mainly from royalties, but said she earned only $13,000 a month from investments. (Her spending, including $102,000 a month for entertainment, gifts and vacation and $16,000 for clothes, was more eye-catching.)

The nature of her investments might be unclear as well. Accountants are trying to learn whether her money was moved into overly risky investment vehicles, whether any business deals need to be undone and whether her assets were outright looted. (Her father filed a report of grand theft, according to The Los Angeles Times, asserting that paintings, jewelry and other valuables had been stripped from her Beverly Hills home).

Apparently, Britney has some shaky assets on her balance sheets. Well, don’t worry Britney. You’re not the only one.

In an announcement that has sent produced a large and varied reaction, the FED has announced that they will attempt to bail out banks by letting them use mortgage-backed securities as collateral for loans. This move is unprecedented in the Fed’s history. For the first time, they are entering the mortgage business. Since its inception, the Fed has used open market operations (the buying and selling of treasury bonds) to expand or contract the monetary policy. A good detailed discussion is here, at interfluidity. Simplistically, the Fed’s balance sheet looks like:

Assets - Liabilities
bonds   -  US currency

When they want to inject liquidity into the market, they buy bonds, paying for them with cash that enters circulation. On the contrary, to contract the money supply, they sell bonds, receiving cash for the sale, and that cash is taken out of the economy. Clearly, their ability to carry out monetary policy depends on the solvency of the bond market. If that market seizes, then they can’t implement monetary policy.

What makes this move by the Fed so remarkable is that they are putting mortgage-backed securities on their assets.  The risk of these securities will therefore be absorbed by the currency, most likely through inflation. Bonds, which are relatively stable assets, compared to ill-maintained houses with bankrupt owners, provide the backing for the currency. This allows some of the risks of inflation to be absorbed. By making the currency more risky, it can drive up expected inflation. The risk comes from several sources

  • the value of the collateral - the mortgage-backed securities - is very volatile.
  • the market for these securities may not be solvent, meaning that if the fed wanted to sell these assets, there may not be buyers

If the fed loses its contract the money supply, this will lead to an inflationary spiral. Additionally, this action, even if it has no immediate negative consequences, it makes people question their expectations of the Fed’s actions. Adding risky assets to the backing of our currency will cause people to divert capital into more stable currencies, such at the Euro, pound, or Real. I had joked with my econ professor that perhaps the day would come that if I couldn’t pay for my car, I could sell it to the Fed. I suppose anything is possible.

Topics: General | 2 Comments »

what I’ve been up to lately

By stephen | March 5, 2008

Hello World,

I know it has been a while. I’ve been busy. There is a lot of great stuff going on in my academic pursuits that I have been wanting to write about for a while. Although I can’t reveal all of the details of the research I have been doing, I do want to talk about some aspects of it.

The main project is….not up for discussion in detail at the moment. :) But, in the process I have used some great tools which I want to share. One of them is Amazon.com’s mechanical turk service.

Mechanical turk is a site where people can post tasks and pay others to complete these tasks. The payments are typically small ($0.01 .. $0.15) and involve repetitive tasks that require human intelligence. Examples include tagging elements in a picture, transcribing a video, reading and summarizing an article, and so forth. It is a great way to have actual humans process a large amount of information for a small amount of money.

In addition to the web interface, they provide programmatic access. I have been using Perl, my scripting language of choice, to work access their API. The general approach is:

  • Select data from a database
  • Generate HITs (Human Intelligence Tasks) by generating an piece of XML conforming to the QuestionForm schema
  • Set a price, expiration time, keywords, and other parameters, and upload the HIT. Submitting a HIT will return to you a HITId, which you store
  • Using the stored HITId, once the task is done, grab the results and put them in a (mysql) database

There are other really cool features of mechanical turk. I plan of a series of mturk articles in the coming week, but needless to say it has allowed us to accomplish things that we could not have done in any other way, or for such a low cost. The development has been really straightforward. I’ve probably written 2500 lines of Perl to do lots of good things for me. Once you understand the mindset/paradigm, the code flows well.

—-

I have also been working with Hadoop.

hadoop

This is a framework that implements the MapReduce algorithm, which is an algorithm distributing the computation of a problem across cluster of computers. MapReduce is designed for problems with many pieces of data, where each piece can be worked on independently of the others. Such problems can be parallelized, with the work being distributed over many computers that are working in a cluster. With Hadoop and MapReduce, one simply writes a Map function, which describes how your data can be broken up, and a reduce function, which aggregates the results. Hadoop handles redundancy and fault-tolerance issues, and provides a distributed file system, HDFS.

A common usage of compute clusters in economics and finance is for Monte Carlo simulations. These are simulations of scenarios, such as stock markets, where there are components of randomness. By running the scenario once, you see one possible outcome. By running the scenario hundreds, millions, or billions of times, you see the distribution of outcomes, given many different scenarios. Hadoop is accessed via a Java API, but is also accessible in C++ via Hadoop Pipes. I am comfortable in either language, so implementation is fun.

Now that we have a great framework, the question is, where do we run our monte carlo simulations? It would be prohibitively expensive for most individuals to maintain their own 100-node cluster of computers. In addition to the hardware costs, there is storage, power, maintenance, and more. And it would not be wise to own such a cluster for the occasional computation. This is where Amazon’s EC2 comes in.

EC2 stands for elastic compute cloud. It allows you to make a cluster of computers that you pay for by the hour. For example, if you want a 5 node cluster for 2 hours, you pay ($0.10 * 5 * 2 = $1.00) for the use of that cluster. You can put whatever operating system you want on the cluster, using either their pre-built configurations or your own custom builds. Clusters can be expanded dynamically and new machines come online within a minute or two. It is really cool.

The images that run on EC2 boxes are called AMIs. One of the public AMIs available on EC2 is a Hadoop image. In fact, running Hadoop on EC2 is very well supported. With a simple command, I can spawn my own cluster of 5, 10, or 20 machines. I can pay for just the time I use, get the results, and be done with the cluster. Essentially, for negligible costs, I have access to an arbitrarily large cluster for an arbitrarily short or long amount of time. How cool is that?

—

To learn some of the asset pricing theory, for which I would be using EC2/Hadoop, I have been reading Steven E. Shreve’s Stochastic Calculus for Finance I: The Binomial Asset Pricing Model. It is extremely well written and I have learned quite a bit. As soon as I am done with this one, I will order these next two:

Eh, em. My wish list is up to date :).

To wrap up, next quarter I have some econometrics, more differential equations stuff, and some small odds and ends that need to get done. The research projects continue, and I am working on teaching myself some Monte Carlo simulations as time permits. I would also like to work on some high frequency data analysis, implemented in C++, but that is on the backburner at the moment.

Like I said, I’ve been busy. The great snow up at the pass isn’t helping my productivity either. Yeah, academic life is busy. The personal life is a whole other issue…. :)

Topics: General | 1 Comment »

my bookshelf

By stephen | January 8, 2008

Hello,

A new quarter has started, and I find myself quite busy. Hopefully I will be able to maintain some interesting posts, but I think this quarter might be a bit sparse. In my study room setup, I have a shelf of books that is close enough to my desk that I can reach them without getting up. So, I can put the important books that I will be using this quarter on this shelf. Here is the status of that shelf.
shelf

I bet you can guess what I am up to this quarter :). It is going to be awesome! Tomorrow I will post my papers reading list.

Topics: Uncategorized | No Comments »

« Previous Entries