Tools of the trade

Tools of the trade
Tools of the trade
In this article I will tell you about the possible tools you have at your disposal for trading. Whether you are building a fence, or building a trading strategy, you need tools to get the work done. In the previous articles we already looked at the question of why, how, what and where to trade. Now it is time to get some tools and setup an infrastructure for trading. We basically need three types of tools.



We need tools for:

  • data
  • analysis
  • execution


Data: get it, munge it, save it

Data: get it, transform it, save it.
Data: get it, transform it, save it.
The first thing we need, is data to use in our analysis, which in turn will lead to the execution of our trades. Not only do we need to know where to get the data, but we often also need / want to transform the data into a (more) usable form, which is called munging of data. And once we have the data and took the effort to transform it somehow, we need to keep / store the data.

Forms of data

Data comes in many forms, shapes and sizes. Mostly we will be looking for:

  • charts
  • historical price data (time series)
  • fundamental data
  • news


So where to get data?

Charts

Price charts are most probably offered by your broker. If you trade FX and CFDs then your broker will most likely offer the MetaTrader platform or maybe the cTrader platform (FX only). Both of these offer good charting in my opinion. If you want to get your charting from another source then your broker, then you have a few options. My favorite option is TradingView. This is a free online charting platform. I have written an article and recorded a youtube video about how to back test strategies using this platform here: http://www.davetromp.net/2015/08/how-to-back-test-manually-traded.html
If you trade or invest in stocks and etfs based on the daily closing prices, then most online finance sites like Google finance, Yahoo finance, Blomberg, etc. offer charting that will do just fine.

Historical Price Data

All charts are nothing more than visual representations of prices over time, also called price time series. This is interesting data to have as it will allow us to analyze prices in a different way, then just visually. Also it will enable the automation of back testing. This will speed up our trading analysis process. Always mind data errors in the freely available data you find online. Especially daily highs and lows can be way off. Therefore I use only closing prices from free online sources. I use the following sources to get historical price data manually:
Stooq tick data
Stooq tick data

  • Yahoo
    Just lookup a security and go to the "Historical Prices" page: http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices
    You can select daily, weekly, monthly and yearly data. Yahoo Finance will give you volume, open, high, low, close prices plus adjusted closing prices. Adjusted closing prices are historical prices that have been adjusted for events like stock splits. For instance, if a stock price was 200 USD adn was split to become 2 times 100 USD alll historical prices before this event would be recalculated by deviding it by 2. This way it is possible to make correct comparison of the prices over time.
  • Google
    Google finance offers more or less the same kind of data as Yahoo Finance. See: http://www.google.com/finance/historical?q=NASDAQ%3AAAPL However they only offer daily prices and do not offer adjusted close prices, so you will have to take this into account when analyzing the data.
  • Stooq
    At www.stooq.com you can download price data for almost any security there is. The site is based in Poland. It is setup in Polish, but the main navigation will translate to English if your browser / os is set to English. I have once contacted their customer support to ask some questions about the data they offer and I got a prompt reply in English. Try that at Yahoo or Google! They offer download options for daily, weekly, monthly, quarterly and yearly data on the historical data page much like Yahoo and Google. For example: http://stooq.com/q/d/?s=eurusd But if you want to get more granular data you can get it as well; even down to tick data. To get this go to the technical analysis page of the security, like: http://stooq.com/q/a/?s=eurusd. This page will load in a java webapplet with a chart of the EURUSD. You can than select any timeframe you like down to 1 minute and ticks. Below the chart there is an option to "Export data to csv file". This is a great option to quickly get some data. However you can only get some data. The only data you can download, is the data that is used for the chart you are looking at. For tick data I am only able to get a 1000 data points and for 1 minute about 2300 data points.
  • Dukascopy
    Dukascopy is a swiss broker. They offer free downloads of complete historical forex price data down to ticks. You do need to register to download. You can find the download options at www.dukascopy.com --> market info --> historical data export.

Getting the data manually can become a drag especially if you want to test a strategy on many different securities. Luckily, there are several services that allow us to easily automate the retrieval of data.

  • Quandl
    Quandl is a service provider that provides access to many data sources, both free and paid. The great thing is that they have different ways to acccess the data. Not only can you simply download the data as csv, json, xml or excel file, they also have libraries for different programming languages that offer easy programmatic access. Check out this example: https://www.quandl.com/data/YAHOO/AAPL If you want to access the data using for instance python and you want to make more than 50 calls a day, then you need to authenticate yourself. You can just signup and get an authtoken.
  • Pandas datareader
    I use python with the pandas library to do many of my analysis and data handling. Pandas has a (now) seperate library that allows easy access to free data from Yahoo, Google and others. Check out https://github.com/pydata/pandas-datareader
  • Oanda api
    My broker Oanda has a rest-api. Among other things it allows for accessing historical price data up to 5000 data points. If you have a practice account you can access the api and retrieve all FX data this way. For access to the CFD data, you need to have a live account that allows for trading CFDs (please note that US customers do not have this possibility). I will go into using the Oanda API to retrieve data in a later post.


How to munge historical price data?

A lot of the times, the data you are getting has errors. So the first thing you want to do after you got the data is to plot it, inspect it visually and adjust / remove any obvious out-layers. You can do this by simply reading in the data into a spreadsheet and make a graph. I do this in LibreOffice Calc or in Python.

Reading in Google Finance data in LibreOffice Calc
Reading in Google Finance data in LibreOffice Calc
Sorting the data in LibreOffice Calc
Sorting the data in LibreOffice Calc
Plotting the data for inspection
Plotting the data for inspection

How to save data, so you can handle it?

I usually just save the data as csv file (comma seperated file), but you can also build a securities database in a SQL database. This way it is easy to reuse the data. You can retrieve and clean the data once and then use it over and over again. Also overtime you will be able to build up a database of data that goes back in time. I will go into how to set this up as I will get to it myself later. I like csv files as they are easy to use and understand. They can be read in into a spreadsheet and a spread sheet can be saved as csv file. I use LibreOffice Calc to do this.
Python with the pandas library. My tool of choice.
Python with the pandas library. My tool of choice.
Also with the Python Pandas library it is very simple to read in a csv file into a dataframe using:
df = pandas.read_csv("file.csv")
The dataframe is more or less like a spreadsheet, but can be manipulated using a few simple lines of Python code. The result can than be saved to a csv file like this:
df.to_csv("newfile.csv")
And than this file can be opened again in LibreOffice Calc or in a text / code editor like Atom. Pandas also has read and write functions for excel, sql, json and others. For me the csv format has been most efficient to use. For large sets of data databases or the HDF5 datastore format might be more convenient.
More info on the read and write options in Pandas can be found here: http://pandas.pydata.org/pandas-docs/stable/io.html

Fundamental Data

I do not use fundamental company data, as I do not trade individual company stocks, but I have looked into stock screeners. Yahoo used to have a very extensive java app, but it does not seem to be available anymore. Google finance has a basic, but pretty good and easy to use screener. See: http://www.google.com/finance#stockscreener Two other interesting sources, that allow for programmatic access to data, including fundamental data are as mentioned above:
  • Quandl.com
  • Pandas datareader

News

I do not trade the new, but I do mind big news events as they may cause higher than normal volatility. Most trading platforms incorporate some sort of news feed. I use Oanda's FxTrade platform and this platform has a nice feature that plots future and past news events on the price charts. This way it is easy to see which events tend to move the markets.
Oanda FxTrade News Events
Oanda FxTrade News Events
Google finance has a similar feature. You can activate news events on charts and than click through to read. Also Tradingview incorporates an economic calendar and an earnings calendar. I do not use these much. For me the FxTrade News feature is enough.

Analysis

Analysis
Analysis
So you got all the data, cleaned it up, now what? Well, have fun with it! With me it all start with simply looking at the data. Our minds are made to look for patterns in randomness. So I use this first, I use my head. The next step is that I try to translate the pattern I think, I have found, into a few simple rules. For example, a rule can be: "if todays closing price is more than 2% higher than yesterdays close price, then the next day will be a down day, meaning I will sell short at the next days open and buy back just before the close.". I write down the rules and then I manually back test the rules using software like TradingView if possible. I than model the rule set in a spreadsheet. The following step it to write a back test in Python. This will allow me to do the testing on more different data samples more quickly, then when I would use only spreadsheets. I you want to read more about why I us Python for this, then go check out this article I wrote:
http://hubpages.com/technology/Learn-python-for-trading-and-investing
Back testing with python also allows for more complicated strategies to be back tested and it is the first step to automated execution of the strategy.

Execution

Execution of your trade
Execution of your trade
The execution of trades can be done in basically three ways:

  1. Through trading platforms
    This is straight forward enough. You just enter your trades in proprietary platforms like Oanda's fxtrade, or cross broker platforms like mt4 and cTrader. Check out my resources page for more info on Brokers and platforms to use.
  2. Through online trading APIs
    Many brokers access via trading APIs. There are different kinds of APIs, but the most commonly used are rest APIs and the FIX protocol api. Restful APIs are apis that are typically used on the web to have web applications interact with each other. The FIX Protocal is a protocol used mainly by professionals to interact with financial platforms like exchanges, brokers and liquidity providers. I have not yet familiarized my self with the protocol, but there is a great Youtube channel FixProtocol with a great series on how to read Fix Protocol messages if you are interested to know more.
  3. Through local API to the trading platform
    Some platforms allow you to hook up excel to the platform, so you can get data into excel from the trading platform and generate trading signals that can than be executed via the trading platform. This means you will have to have everything on your computer or on a server. Some platforms also allow for programmatic access. For instance, cTrader has cAlgo that allows you to program your strategies in cSharp and InterActive Brokers has a Java based API.

I use python wrappers to access the Oanda Rest API and if I would want to access the InterActive Brokers trading platform jave based API I could use ibpy. This highly simplifies accessing the APIs and you do not have to re-invent the wheel. Github is your friend here:
https://github.com/blampe/IbPy
https://github.com/oanda/oandapy

I know I only briefly touched on many different things and I referenced a lot of sources. My intention is now to get you familiarized with the tool I use and to get you researching all the tools I reference. In later articles I will be going more deeply into each tool with examples of how I use them precisely.

Next in this getting started trading series

If you like this "getting started with trading" series and wanna stay in the loop on new articles, then be sure to subscribe using the social media buttons and email form below this post.

Comments