Tools of the trade
|Tools of the trade|
We need tools for:
Data: get it, munge it, save it
|Data: get it, transform it, save it.|
Forms of dataData comes in many forms, shapes and sizes. Mostly we will be looking for:
- historical price data (time series)
- fundamental data
So where to get data?
ChartsPrice charts are most probably offered by your broker. If you trade FX and CFDs then your broker will most likely offer the MetaTrader platform or maybe the cTrader platform (FX only). Both of these offer good charting in my opinion. If you want to get your charting from another source then your broker, then you have a few options. My favorite option is TradingView. This is a free online charting platform. I have written an article and recorded a youtube video about how to back test strategies using this platform here: http://www.davetromp.net/2015/08/how-to-back-test-manually-traded.html
If you trade or invest in stocks and etfs based on the daily closing prices, then most online finance sites like Google finance, Yahoo finance, Blomberg, etc. offer charting that will do just fine.
Historical Price DataAll charts are nothing more than visual representations of prices over time, also called price time series. This is interesting data to have as it will allow us to analyze prices in a different way, then just visually. Also it will enable the automation of back testing. This will speed up our trading analysis process. Always mind data errors in the freely available data you find online. Especially daily highs and lows can be way off. Therefore I use only closing prices from free online sources. I use the following sources to get historical price data manually:
|Stooq tick data|
Just lookup a security and go to the "Historical Prices" page: http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices
You can select daily, weekly, monthly and yearly data. Yahoo Finance will give you volume, open, high, low, close prices plus adjusted closing prices. Adjusted closing prices are historical prices that have been adjusted for events like stock splits. For instance, if a stock price was 200 USD adn was split to become 2 times 100 USD alll historical prices before this event would be recalculated by deviding it by 2. This way it is possible to make correct comparison of the prices over time.
Google finance offers more or less the same kind of data as Yahoo Finance. See: http://www.google.com/finance/historical?q=NASDAQ%3AAAPL However they only offer daily prices and do not offer adjusted close prices, so you will have to take this into account when analyzing the data.
At www.stooq.com you can download price data for almost any security there is. The site is based in Poland. It is setup in Polish, but the main navigation will translate to English if your browser / os is set to English. I have once contacted their customer support to ask some questions about the data they offer and I got a prompt reply in English. Try that at Yahoo or Google! They offer download options for daily, weekly, monthly, quarterly and yearly data on the historical data page much like Yahoo and Google. For example: http://stooq.com/q/d/?s=eurusd But if you want to get more granular data you can get it as well; even down to tick data. To get this go to the technical analysis page of the security, like: http://stooq.com/q/a/?s=eurusd. This page will load in a java webapplet with a chart of the EURUSD. You can than select any timeframe you like down to 1 minute and ticks. Below the chart there is an option to "Export data to csv file". This is a great option to quickly get some data. However you can only get some data. The only data you can download, is the data that is used for the chart you are looking at. For tick data I am only able to get a 1000 data points and for 1 minute about 2300 data points.
Dukascopy is a swiss broker. They offer free downloads of complete historical forex price data down to ticks. You do need to register to download. You can find the download options at www.dukascopy.com --> market info --> historical data export.
Getting the data manually can become a drag especially if you want to test a strategy on many different securities. Luckily, there are several services that allow us to easily automate the retrieval of data.
Quandl is a service provider that provides access to many data sources, both free and paid. The great thing is that they have different ways to acccess the data. Not only can you simply download the data as csv, json, xml or excel file, they also have libraries for different programming languages that offer easy programmatic access. Check out this example: https://www.quandl.com/data/YAHOO/AAPL If you want to access the data using for instance python and you want to make more than 50 calls a day, then you need to authenticate yourself. You can just signup and get an authtoken.
- Pandas datareader
I use python with the pandas library to do many of my analysis and data handling. Pandas has a (now) seperate library that allows easy access to free data from Yahoo, Google and others. Check out https://github.com/pydata/pandas-datareader
- Oanda api
My broker Oanda has a rest-api. Among other things it allows for accessing historical price data up to 5000 data points. If you have a practice account you can access the api and retrieve all FX data this way. For access to the CFD data, you need to have a live account that allows for trading CFDs (please note that US customers do not have this possibility). I will go into using the Oanda API to retrieve data in a later post.
How to munge historical price data?A lot of the times, the data you are getting has errors. So the first thing you want to do after you got the data is to plot it, inspect it visually and adjust / remove any obvious out-layers. You can do this by simply reading in the data into a spreadsheet and make a graph. I do this in LibreOffice Calc or in Python.
|Reading in Google Finance data in LibreOffice Calc|
|Sorting the data in LibreOffice Calc|
|Plotting the data for inspection|
How to save data, so you can handle it?I usually just save the data as csv file (comma seperated file), but you can also build a securities database in a SQL database. This way it is easy to reuse the data. You can retrieve and clean the data once and then use it over and over again. Also overtime you will be able to build up a database of data that goes back in time. I will go into how to set this up as I will get to it myself later. I like csv files as they are easy to use and understand. They can be read in into a spreadsheet and a spread sheet can be saved as csv file. I use LibreOffice Calc to do this.
|Python with the pandas library. My tool of choice.|
df = pandas.read_csv("file.csv")The dataframe is more or less like a spreadsheet, but can be manipulated using a few simple lines of Python code. The result can than be saved to a csv file like this:
df.to_csv("newfile.csv")And than this file can be opened again in LibreOffice Calc or in a text / code editor like Atom. Pandas also has read and write functions for excel, sql, json and others. For me the csv format has been most efficient to use. For large sets of data databases or the HDF5 datastore format might be more convenient.
More info on the read and write options in Pandas can be found here: http://pandas.pydata.org/pandas-docs/stable/io.html
Fundamental DataI do not use fundamental company data, as I do not trade individual company stocks, but I have looked into stock screeners. Yahoo used to have a very extensive java app, but it does not seem to be available anymore. Google finance has a basic, but pretty good and easy to use screener. See: http://www.google.com/finance#stockscreener Two other interesting sources, that allow for programmatic access to data, including fundamental data are as mentioned above:
- Pandas datareader
NewsI do not trade the new, but I do mind big news events as they may cause higher than normal volatility. Most trading platforms incorporate some sort of news feed. I use Oanda's FxTrade platform and this platform has a nice feature that plots future and past news events on the price charts. This way it is easy to see which events tend to move the markets.
|Oanda FxTrade News Events|
|Execution of your trade|
- Through trading platforms
This is straight forward enough. You just enter your trades in proprietary platforms like Oanda's fxtrade, or cross broker platforms like mt4 and cTrader. Check out my resources page for more info on Brokers and platforms to use.
- Through online trading APIs
Many brokers access via trading APIs. There are different kinds of APIs, but the most commonly used are rest APIs and the FIX protocol api. Restful APIs are apis that are typically used on the web to have web applications interact with each other. The FIX Protocal is a protocol used mainly by professionals to interact with financial platforms like exchanges, brokers and liquidity providers. I have not yet familiarized my self with the protocol, but there is a great Youtube channel FixProtocol with a great series on how to read Fix Protocol messages if you are interested to know more.
- Through local API to the trading platform
Some platforms allow you to hook up excel to the platform, so you can get data into excel from the trading platform and generate trading signals that can than be executed via the trading platform. This means you will have to have everything on your computer or on a server. Some platforms also allow for programmatic access. For instance, cTrader has cAlgo that allows you to program your strategies in cSharp and InterActive Brokers has a Java based API.
I use python wrappers to access the Oanda Rest API and if I would want to access the InterActive Brokers trading platform jave based API I could use ibpy. This highly simplifies accessing the APIs and you do not have to re-invent the wheel. Github is your friend here:
I know I only briefly touched on many different things and I referenced a lot of sources. My intention is now to get you familiarized with the tool I use and to get you researching all the tools I reference. In later articles I will be going more deeply into each tool with examples of how I use them precisely.
Next in this getting started trading series
If you like this "getting started with trading" series and wanna stay in the loop on new articles, then be sure to subscribe using the social media buttons and email form below this post.