Improved Portfolio Optimizer, and Initial Results

/0 Comments/in /by

I’ve been working hard on making the tools better, in particular on the optimizer, to make sure it produces great results. Keep in mind that studies that use historical returns can only be relied on so much. And we should never be using historical results to tweak the portfolio, because it leads to overfitting. If you are not aware, overfitting is a data science term that refers to making the model ‘fit’ the data. Like fitting for a suit of clothes. The problem is, that market performance won’t be duplicated, so the model might promptly fail on new ‘real’ results.

So with that said, I’ve improved the optimizer so even with smaller portfolios it produces better results. The optimizer is about reducing risk, and when it results in better returns and reduced risk, that’s a nice result. (again with the caveat I made about past results)

A couple weeks ago I made a blog post about optimizing some popular model portfolios from a Forbes article. At the time I thought it would be interesting to try optimizing these portfolios using RIAengine smart portfolio optimizer, and seeing what backtests looked like.

Now, with the improvements to the optimizer, I wanted to post updated results.

Buffet 90/10 Portfolio

This portfolio optimization had resulted in better risk and drawdowns, but returns were worse. Here are the new results:

So like with the original tests I ran, I optimized for 2003 and 2004, and then ran a backtest from 2005 through 2010. You don’t want to optimize and backtest over the same period since you’ll be overfitting. So that’s why the sequence there.

The original tests had resulted in worse performance, but improved risk and drawdowns. And the new test, with the improved optimizer, still resulted in better risk and drawdowns, but performance was higher too. This of course would be a best of both worlds scenario. Kind of a surprising result with only two assets to optimize for, but interesting too.

Merriman Ultimate Buy and Hold Strategy

The next model, the Merriman Ultimate Buy and Hold Strategy invests pretty much all in stocks, some large, some small, some value, some blend. Some are US and some international. Last, a little real estate gets added. As I said before, you need different asset classes to properly diversify. But let’s see how it goes.

The last time I ran these tests, the optimizer produced a worse backtest than the original. The drawdowns were worse, risk about the same, and returns worse. This time, returns still less but at least they are positive. Risk still the same and drawdowns worse.

As I said before, with an almost 100% stock portfolio, and only real estate to diversify things, it’s hard to figure out an asset mix that will perform better than any other. Ideally you’d reduce the drawdowns primarily.

Just for kicks, I decided to try increasing the real estate by a lot and see if it improves the drawdowns. So I increased them to 30% of the overall portfolio.

As you can see, it’s not really an improvement. Drawdowns still over 60%. If drawdowns are 60%, you need returns of 150% to get back to even! So, probably need to avoid this portfolio. Sorry Merriman.

Once again, these are hypothetical results, based on historical returns. And future performance won’t ever be exactly like any past period.

Ivy League Endowments portfolio

Next up, the Ivy League Endowments. You are probably getting used to this, so let’s jump into the results:

So the result is that the improved optimizer resulted in better returns, while risk is still better than the original portfolio, and drawdowns as well. So a good result.

Generally speaking, things that improve returns will probably result in more risk, but not always. We’re shooting for both, but if we have to choose, lower risk is preferable.

As I’ve said before, the risk of the client losing their stomach in the market is worse than just drawdowns!

Coffeehouse Portfolio

The next portfolio was called Coffeehouse and is a good mix of assets. Let’s see what happens with the comparison:

The improved optimizer gave a much better result with this model and backtest. Returns are much better, though a little less than the original. But risk, sharpe, drawdowns, all much better. While risk, sharpe and drawdowns are not as good as the first time we optimized, overall the result is more balanced.

Bill Bernstein’s No Brainer Portfolio

This portfolio is just 4 assets, equally weighted. The optimizer pushed the bonds higher than stocks, and lowered the real estate allocation. Let’s see how it turned out.

As expected, with more bonds, returns should be less normally, while risk should also be less. But since the improved optimizer didn’t allocate so much to bonds as the first time we ran this, the effect was less. This means we have higher returns than before, with still improvement in risk measures. On to the last portfolio.

Harry Browne’s Permanent Portfolio

Like the Bernstein portfolio, this one allocates equally to 4 securities, but it includes gold rather than real estate. The results:

The results are similar to the Bernstein portfolio, and similar compared to the last test. Returns are improved over the last test, though still not as good as the original models. However, once again a nice balanced result. Still reducing the risk of the original portfolio, and less returns but not dramatically so. Overall the client would sleep better at night.

Takeaways And Notes

This time around, results are more balanced, so the improvements are looking promising. Once again, past performance won’t be repeated. With portfolio optimization, we’re looking for diversification that results in lower risk portfolios, and trying not to reduce returns. But we hate drawdowns, since they make it harder to recover. A portfolio that’s underwater for a long time results in an unhappy client!

Machine Learning In Investing

/0 Comments/in , , , , , /by

machine learning in investingMachine learning can be used in investing in a variety of ways. This article seeks to show a few of those and describe the basics.

Hedge funds have tried for years to use machine learning (or AI) in their funds. Moving on to today, all quantitative funds use some form of machine learning. Some hedge funds even build around machine learning specifically, spurning human intervention.

Risk Management

Risk usually means standard deviation and positive correlation. However machine learning can approach risk management in a different way. Traditional investment correlation studies the way stocks move together. Machine learning today can look at correlation of risky investments with alternative data points. For example a model can look at how risk of a certain industry’s securities might rise and fall with weather data points, such as rain or temperature.

Simple methods of machine learning can be used to reduce the prediction errors in mean variance portfolio optimization. This helps manage the risk that can be reduced via diversification.

Building Portfolios

When discussing machine learning in investing, it’s natural to think about building portfolios. Since machine learning can assist with portfolio optimization, it can also be used to select investments to make up ideal portfolios.

Price Prediction

Of all the ways to use machine learning in investing, this one has gotten the most attention. This is because everyone is looking for the holy grail, also known as ‘the algorithm that accurately says when to buy and sell a security’. Many quant platforms are built around this concept.

Macro Economic Predictions

In addition to predicting prices, machine learning is also used to predict the state of the economy. Portfolio optimization depends quite a lot on return estimates, and return estimates are only as good as predicted economic conditions. Traditionally, to figure out what the economy was going to do, you would look at oil, interest rates, or similar macro data points. Now, you can throw huge amounts of alternative data at a model designed to look for similarities in past economic periods to current conditions. These models reinforce the estimates more than can be done with simple machine learning. They should result in much better estimates.

A word about Data

In order to use more complicated machine learning models, you need a lot of data. How much, that depends. It’s safe to say that the starting point to do meaningful things with deep learning, is at least a couple terabytes.

Optimizing Some Popular Portfolios

/1 Comment/in /by

In Forbes I found a post about some ‘expert’ model portfolios. I thought it would be interesting to try optimizing these portfolios using RIAengine smart portfolio optimizer, and seeing what backtests looked like. Here are the results.

Buffet 90/10 Portfolio

The first portfolio is simple, just 90% allocation to an S&P500 index fund (I chose SPY) and a 10% allocation to Bonds, I used SHY. Here is how it went:

Optimized Buffet Portfolio

What happened is the portfolio went from 90% SPY to 98% SHY. We ran a backtest on it using RIAengine backtester, and tested over 5 years including the crash of 2008. This resulted in a portfolio that returned a little less, but had dramatically less risk. It’s about what you’d expect going from mostly stocks to mostly bonds. Sharpe ratios are both negative since neither portfolio outran the risk free rate. While interesting, the optimizer is meant to handle more instruments than 2 in order to work properly.

Merriman Ultimate Buy and Hold Strategy

Let’s move on to the next portfolio in the article, the Merriman Ultimate Buy and Hold Strategy.This portfolio invests pretty much all in stocks, some large, some small, some value, some blend. Some are US and some international. Last, a little real estate gets added. Right off the bat, without some other asset class in there it will be hard to properly diversify. But let’s have a go.

Optimized Merriman Portfolio

If you want an optimal portfolio, you better have different asset classes to work with!

The interesting result is that sitting all in stocks you will have a hard time overcoming drawdowns. In this case, both portfolios had drawdowns of over 60%. To overcome this kind of drawdown your portfolio has to earn 250%! That’s why I like to talk about avoiding drawdowns, instead of maximizing returns. And sharpe ratios are negative because the portfolios didn’t earn more than the risk free rate of around 3%.

Good lesson here is get some more asset classes when optimizing a portfolio so you can take advantage of diversification.

Ivy League Endowments portfolio

Next up, the Ivy League Endowments portfolio. When optimizing a portfolio it’s good to have different asset classes to work with. This one has large cap stocks, bonds, real estate, and it should have some commodities. Since I couldn’t find commodity ETFs that have been around since 2003, I used an energy ETF, IYE. Here are the results:

Optimized Ivy League Endowments portfolio

The optimized portfolio ended up with significantly less risk, though it made half the gains. Drawdowns are lower by 38%. This will help the client hold on when things are going the wrong way. The risk of a 45% drawdown is more than just market risk. The real risk is the client can’t stomach it.

Coffeehouse Portfolio

This portfolio has a lot of equity, and add some bonds and real estate. The results:

Neither portfolio outran the risk free rate. The optimal portfolio had lower risk and drawdowns.

Bill Bernstein’s and Harry Browne’s Portfolios

The last 2 portfolios are similar in that they have 4 assets equally weighted. The results of optimizing these portfolios is a flight into lower risk, leading to much lower drawdowns and also returns.

Takeaways And Notes

All of these tests were run with no constraints or limits on asset weights. Quite often this leads to the lower risk assets getting the lion’s share of overall weight. So it’s usually best to set constraints to balance the results.

Another thing to note is that backtesting is not research. If you use the backtest to help you tweak parameters such as asset weights in optimizing a portfolio, it will almost certainly lead to misleading results. This is because you start to fit the backtest to the data. In this case I’m optimizing the portfolio from 2003 through 2004 (two years of returns is minimum) and running backtests from 2005 through 2010. If I keep tweaking the results so that the backtest performs well from 2005 through 2010, then I’m overfitting to the data.

So use the backtest to get baseline information, but don’t go too far with your tweaking!

RIAengine will be launching the smart portfolio optimizer soon to a small beta group before launching to the public. Sign up to be notified of the launch.

Investment Portfolio Risk, Reduce It Or Risk Losing Clients

/1 Comment/in /by

Investment Portfolio RiskInvestment portfolio risk can be an uncomfortable topic.

It’s easy to find articles all over the internet that encourage people to sit tight when things are dropping. Warren Buffet’s famous instructions regarding his wife’s trust are to put 90% of it in the S&P500. And since Warren Buffet says to do it, a lot of people think we should all do the same. (side note, with all that money invested, or course Warren Buffet wants others to sit tight! He’s a rational investor, not a charitable organization!)

But is it working? According to Dalbar, the research firm, while the stock market in general has returned just under 10% a year historically, the average investor only earns over 5%! Why is that? The reason comes down to investment portfolio risk. They don’t enjoy the ride as much as the hoped-for destination. So when the portfolio keeps dropping, they begin to doubt they will ever arrive.

So they call you, the advisor. If you are like most financial advisors, you will coach them on hanging in there, after all, the portfolio is designed to help them meet their future goals. But what if they are just uncomfortable with that amount of risk? Then you might lose them with your message, or lose them as a client.

Perhaps a question to ask yourself is, are your model portfolios too risky?

So How Can Your Reduce Investment Portfolio Risk?

Whether you have model portfolios for each individual, or a set of 5 or 10 that you apply based on risk profiles, you can find ways to reduce portfolio risk.

First, when you built the portfolios, you may not have optimized them properly, leading to a lessening of the benefits of diversification. This leads to greater risk than necessary, which leads to uncomfortable clients.

Next,  your allocations might not effectively minimize the risk of the investments you’ve chosen. This is not the same as the first point: This isn’t about asset allocations, rather allocations to individual investments to meet those allocations.

And last, there may be other investments that will, by adding them to the portfolio, reduce risk while again not making a big impact on expected returns.

Limiting Drawdowns Rather Than Maximizing Returns

/0 Comments/in , , /by

Prominent asset manager Mary Meeker once said her primary goal is limiting drawdowns! While most portfolio construction is explicitly or implicitly focused on returns, she knew that minimizing events that can decimate the portfolio is more important for long term gains than trying to maximize growth periods. Famously, a portfolio needs to double to make up for a 50% drawdown.

So how to minimize drawdowns?

Portfolio diversification will naturally make a difference in drawdowns, as long as it’s actually diversified. The problem with diversification is in order to try to come to the best asset weights, you have to subject it to some process of optimization. And many optimization methods are known to reduce the benefits of diversification either because the asset mix isn’t linked or due to estimation errors on the front end.

So using robust optimization methods is key.

These robust methods lead to a portfolio with less risk and volatility, and therefore less risk of drawdowns.

For example, taking a portfolio made up of securities randomly selected from 2000-2010 and run through typical covariance/correlation matrices, here is what a graph of the correlation matrix looks like:


The yellow diagonal is a correlation of 1, or every asset with itself. Then the blocks move from yellow to dark blue as the correlation between them is less or negative.

Now let’s re-sort the assets by their correlations so that the highest are on one end and lowest on the other. This results in a graph with a much more orderly appearance:





In this graph, the assets are now clustered and it looks like a heat map, with hot being high correlation.

So with the assets being more closely linked to those that are ‘corr-a-likes’, when we perform our optimization we end up with a set of weights that will have lower risk due to diversification having been preserved.

This lower risk portfolio results in lower drawdowns. And often higher returns, particularly during more challenging stretches in the market.

RIAengine will be launching a machine learning portfolio optimizer to a small beta group within a couple weeks.

Questions or comments? Love to hear them, please leave a comment below.

Machine Learning in Finance and Investing

/2 Comments/in , , , /by

The first time I heard the term ‘Artificial Intelligence’ was in the 1980s, when my dad worked for Digital Equipment Corporation, in the Silicon Valley of the East (Massachusetts). And yes they made computer chips from Silicon. In the 80’s Artificial Intelligence referred to, basically, complex logical structures, with each decision leading to a tree of subsequent possibilities. It’s all compute power allowed in the 80’s and 90’s.

Fast forward to where we are today, and compute power is vastly greater, and machine learning can now avail itself of that power. Now instead of advanced decision trees, machine learning experts are trying to mimic the human brain.

AI (Artificial Intelligence) and ML (Machine Learning) tools abound today, in every industry it seems. Netflix uses it to figure out their users, as does Amazon, Google, and Apple, and Uber uses it to optimize drivers and routes for passengers. Political candidates try to use it to gain an edge on other candidates. It’s literally everywhere.

In finance, it’s being used for evaluating credit risk, make suggestions for financial planning goals, and even for investments.

Hedge funds have used machine learning for years, poring over many petabytes of data to find opportunities. They spend billions of dollars every year on data, because without great and unique data sets they can’t gain that edge that allows them to gain 30%, 50% or more every year (the best hedge funds earn over 70%).

But machine learning shouldn’t be limited to the ultra privileged at the top of the financial food chain.

We can use machine learning to make incremental improvements in our portfolios. In the same way it’s common knowledge that high investment fees erode a portfolio over time, making a dramatic difference in the end result, so too incremental improvements in a portfolio, both on the risk side and returns, can make a dramatic difference in your client being able to reach the end goal.

And even a portfolio earning the same returns with less risk means a client who isn’t constantly being bullied by the market. You can help them sleep better at night which adds to your bottom line in the end.

RIAengine will be launching a machine learning portfolio optimizer to a small beta group within a month.

Questions or comments? Love to hear them, please leave a comment below.

How To Best Evaluate A Portfolio

/0 Comments/in , , , /by

Recently I saw an article about Monte Carlo Analysis, which (as you know) is a way to evaluate a person’s ability to meet their future income needs. Using a portfolio of securities or asset allocations it runs through as many rolling periods as available in the data, to come up with a probability of meeting future income needs.

Certainly Monte Carlo Analysis is not great with all the input of expectations, almost certainly it will give misleading results: “OK Client, you are 87% likely to meet your future income needs based on this plan!” But is that percentage really true?.

The article says it’s better to focus on ambiguity rather than just investment risk.

So can Monte Carlo Analysis be Improved Upon?

Monte Carlo has a lot of uncertainty built into it, from all the ambiguous client expectations through the uncertainty about future markets.

One way to improve upon Monte Carlo Analysis would be to incorporate better analysis of coming market cycles: Using machine learning to look at factors such as interest rates, commodities, global events, and stock market risk, we could place higher weights on Monte Carlo periods that began when characteristics were similar to the present. Rather than looking at every period as equally likely to occur, we’d be placing higher weight on the ones that our analysis determined to be more likely to occur.

The downside would be that this would introduce yet another set of expectations.

So an even more solid approach is to tweak the portfolio such that it’s more likely to achieve needed returns without unnecessary risk. By using the Hierarchical Risk Parity approach, we can optimize the portfolio without obliterating the benefits of diversification.

RIAengine will be launching a better portfolio optimizer to a small beta group within a month.

Questions or comments? Love to hear them, please leave a comment below.

A Better Portfolio Optimization Method

/7 Comments/in , , , /by

You may have heard of Markowitz’s Critical Line Algorithm, or CLA. CLA was Markowitz’s attempt to solve the problem of portfolio optimization using quadratic math. For a couple reasons is was an ingenious invention.

But it also has its problems.

The problem with CLA is that it demonstrates large changes in the portfolio with small changes in the forecasted returns of assets. This is a big problem that seems intuitive, but the widespread influence of CLA shows that people are not universally aware of a solution to this problem. While the Critical Line Algorithm isn’t used universally, its implications are everywhere in finance in the construction of portfolios.

So what causes these flaws?

According to Marcos Lopez de Prado in Advances in Financial Machine Learning, the problem with CLA is attributable to covariance matrices basically being unaware of asset classes. In other words, when the portfolio assets are calculated in a covariance matrix to determine whether the assets move together or against each other, the assets are all treated the same, as if they are each replaceable with any other of them. But the assets were added by asset classes and you would not consider replacing a large cap stock with a small cap stock if your portfolio approach said you need large caps in the portfolio. But you might replace a large cap stock with another large cap. So those two assets should be considered linked.

With CLA, the more you diversify, the more likely you have estimation errors in the resulting portfolio. This could entirely negate the benefits of diversification!

So using the covariance result, more effort is needed to keep like assets linked. How to do this? Using machine learning, we can cluster assets with like assets and not lose their linkage when optimizing allocations.

But will this work?

As with traditional portfolio optimization approaches, the covariance matrix should still be used. But instead of relying on its results in raw form, we can build the results into a hierarchical structure which is consistent with how most portfolio construction begins anyway. According to De Prado, now the head of machine learning for AQR Capital, a $226 Billion fund, this results in a portfolio optimization that generates less risky portfolios compared to traditional risk parity methods.

For RIAs, it’s very important to manage your data flow

/0 Comments/in /by

I know, “data flow” sounds a little pretentious, and what the heck does it mean?

The core of your business works between several key applications: a financial planning application, a portfolio management system, of course a CRM, and maybe one or two others. 

So what happens is client data needs to go from one thing to the next. For most advisors, this leads to multiple efforts to collect data from the client, and manual entry of data into each application, which can take hours for each one.

This is poor data flow.

Better would be a tool to collect data from your clients with the end in mind. The end being generating a financial plan, and a portfolio, and whatever else your practice requires.

This would be a major improvement in data flow, and a major improvement to your workflows and save you hours of time every week.

Upgrading Postgres to the latest version on Centos 7 Server

/1 Comment/in , , /by

I recently upgraded postgres to the latest version and I think the steps can help someone else too. So here they are. This assumes Centos 7 server installed.

First, take a backup of database

First SSH into the server.

Now connect to postgres:
sudo -i -u postgres

Now do a dump of the entire install:
pg_dumpall > outputfile

I usually use all_db.sql as the filename, but you can use whatever.

Since the file gets dropped into the pgsql folder, good to move it out of there into your home directory, since we are removing old versions of postgres. Use CTRL^d to get out of postgres.

Then do a mv:
sudo mv var/bin/pgsql/all_db.sql all_db.sql

Make sure the file is in your home folder:
ls

Does the file get listed? Good if yes. If not then you didn’t follow the above exactly.

Next, remove postgres

Next remove all installs of postgres:
sudo yum -y remove postgres\*

Now Install PG latest version

Right now postgres 9.6 is the latest version, but in the future you can follow the same steps replacing 96 or 9.6 with 97 or 9.7 etc.

First add the latest version to your rpm for install using yum. Find latest packages here:

https://yum.postgresql.org/repopackages.php

Copy the link for the OS you have.

screen-shot-2016-10-21-at-1-16-22-pm

…in this case I have centos 7. So I copied this link:

https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm

Now build the rpm command by changing
https://download.postgresql.org/pub/repos/yum

to

http://yum.postgresql.org

… in the next command.

Now run it:
sudo rpm -Uvh http://yum.postgresql.org/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm

Then run the install command:
sudo yum install postgresql96-server postgresql96

Initialize with this:
sudo /usr/pgsql-9.6/bin/postgresql96-setup initdb

Start postgres:
sudo systemctl start postgresql-9.6
sudo systemctl enable postgresql-9.6

Change Configuration Files

If this install of postgres needs to be accessible to the outside world, you can open it up. We’re using firewall rules on the server which opens port 5432 only to certain IP addresses. So without that additional step, you would want to be careful with the below.

Find the configuration file.
sudo -i -u postgres
psql
SHOW config_file;

The location of the configuration file is listed now. So open it for editing. (use your location, the below is mine)

sudo vim var/lib/pgsql/9.6/data/postgresql.conf

Scroll to “CONNECTIONS AND AUTHENTICATION” section and find this line:
#listen_addresses = 'localhost' # what IP address(es) to listen on;

Edit this line as follows:
listen_addresses = '*' # what IP address(es) to listen on;

Exit the file saving changes.
ESC
:wq

Next edit the pg_hba.conf file, in the same folder:
sudo vim var/lib/pgsql/9.6/data/pg_hba.conf

Scroll to the bottom of the file and add these lines if they don’t already exist:
#IPv4 remote connections (all users and IP addresses):
host all all 0.0.0.0/0 md5

On the second line beginning with ‘host’ make sure there is no # added. You want it to read as above.

Exit saving changes.

Now restart postgres.
sudo service postgresql-9.6 restart

Add users

If you had different users setup in postgres, add them again.

CREATE USER user_name WITH PASSWORD 'pass_word';

Replace user_name and pass_word with yours.

Make the user a superuser if that’s what you want.

ALTER USER user_name WITH SUPERUSER;

All done!

Mentions: Thanks to http://tecadmin.net/install-postgresql-9-5-on-centos/ for help with the rpm and install steps.