How to download free OHLC historical cryptocurrency daily data using a python script

<p>With bitcoin recently reaching above the 11,000 USD per BTC mark and other cryptocurrencies such as ethereum gaining traction as well, it has now become common place for people to start developing trading systems for the cryptocurrency space. One issue with this however is that cryptocurrency data is by no means centralized – like regular Forex data – which makes it difficult for traders to get a good historical data source. Many of the data repositories that are available online are not free – charging even thousands of dollars for data – while others contain data that is simply not updated regularly (such as <a href="https://www.kaggle.com/mczielinski/bitcoin-historical-data">this kaggle data source</a>). Today I am going to talk about an easy and reliable way to get cryptocurrency data, allowing you to update your data whenever you want and get access to years of daily historical data for many different cryptocurrencies and exchanges in only a few seconds.</p>
<p>–</p><pre>import numpy as np
import datetime
import time
import re
import pandas as pd
import requests
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('-e', '–exchange')
parser.add_argument('-p','–pair')
args = parser.parse_args()

pair = args.pair
exchange = args.exchange

resp = requests.get("https://api.cryptowat.ch/markets/{}/{}/ohlc?after=1000000000&amp;periods=86400".format(exchange, pair))
resp = resp.json()
resp = resp["result"]["86400"]

all_candles = []
for c in resp:
all_candles.append({"Date": time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime(c[0])), "Open": float(c[1]), "High": float(c[2]), "Low": float(c[3]), "Close": float(c[4]), "Volume": float(c[5])})

df = pd.DataFrame(all_candles)
df = df.set_index(["Date"])
df = df[["Open", "High", "Low", "Close", "Volume"]]
print df
df.to_csv("{}_{}_1440.csv".format(exchange, pair))</pre><p>–</p>
<p>Although different exchanges offer interfaces that allow us to download data from each one of them it is quite cumbersome to implement a separate solution to download data from each different cryptocurrency exchange. Thankfully there is a website called <a href="https://cryptowat.ch/">cryptowatch</a> which has pooled cryptocurrency data from several different exchanges into an easy to use resource. Even better, cryptowatch offers a<a href="https://www.google.com.co/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwi8tKD03u7XAhXELyYKHddmBnYQFggmMAA&amp;url=https%3A%2F%2Fcryptowat.ch%2Fdocs%2Fapi&amp;usg=AOvVaw30L4WgJINsEbGfW7gfEHap"> REST API</a> that allows us to easily access their service in order to obtain historical data and general cryptocurrency information (even order-book data). Today we are going to be taking advantage of their OHLC request function in order to download historical data from their servers from any particular broker that interests us.</p>
<p>The script I have posted above allows you to easily obtain cryptocurrency OHLC daily data for any exchange/pair you want that is carried by cryptowatch. The python script needs to be executed with the parameters “-p” (pair) and “-e” (exchange) which tell it exactly what exchange/pair combination you desire. For example running the above scripts by using the command “python script.py -p btcusd -e kraken” will download all available historical data for bitcoin (for kraken this is from 2013 to 2017), the data is then printed on the terminal and also saved into a csv file for future loading/use without having to call the servers again. <strong>The OHLC prices seem to be Bid prices and the time zone of the candles is GMT 0:00. </strong></p>
<p>–</p>
<p><img loading="lazy" class="aligncenter wp-image-5916" src="https://mechanicalforex.com/wp-content/uploads/2017/12/Selection_999984.jpg" alt="" width="567" height="374" srcset="https://mechanicalforex.com/wp-content/uploads/2017/12/Selection_999984.jpg 747w, https://mechanicalforex.com/wp-content/uploads/2017/12/Selection_999984-300×198.jpg 300w" sizes="(max-width: 567px) 100vw, 567px" /></p>
<p>–</p>
<p>Use of this script has some limits as well, when using the cryptowatch servers you’re limited to a given amount of server time per hour, so it is a good idea to avoid calling scripts that request a lot of data frequently. If you want to update your data more frequently then you should modify the API calls and dataframe processing to append and modify your already downloaded full history file instead of having to redownload the entire historical data set each time. It is also worth noting that you can also request candles for other timeframes if you wish, but you will have to modify the calls to do this (you can refer to the cryptowatch documentation for more information about the parameter calls for this API functions). For lower timeframes the total amount of data available will be less and – if the dataframe is too small – you will probably need several hours and calls structured with the before/after parameters to properly get all the data in chunks.</p>
<p>It is also worth mentioning that although there are several python libraries to interface with the cryptowatch API – such as <a href="https://github.com/astew/cryptowatch-python">this one</a> and <a href="https://github.com/alexanderepstein/cryptowatch">this one</a> – I couldn’t get any of these libraries to work correctly as their data processing seemed to always give me incorrect times when asking for OHLC data, probably due to some issues with how these libraries parse the return OHLC values. The easiest way I found to get the data was to actually write the REST API call myself and to then parse the returned json values manually to put them into a pandas dataframe. Besides showing you how to download the data the script also shows you how to get the data into a pandas dataframe, a very useful thing if you want to perform some trading or data analysis in python using these data sets.</p>
<p>–</p>
<p><img loading="lazy" class="aligncenter wp-image-5917" src="https://mechanicalforex.com/wp-content/uploads/2017/12/Selection_999985.jpg" alt="" width="474" height="349" srcset="https://mechanicalforex.com/wp-content/uploads/2017/12/Selection_999985.jpg 603w, https://mechanicalforex.com/wp-content/uploads/2017/12/Selection_999985-300×221.jpg 300w" sizes="(max-width: 474px) 100vw, 474px" /></p>
<p>–</p>
<p>The above should give anyone interested in the design of cryptocurrency trading systems – or even just the analysis of cryptocurrency data – a good start. The first problem with system development is always finding some historical data, once this is solved we can start working with it. Whether something useful can be done with it is an entirely different matter altogether. If you would like to learn more about how we design systems and how you too can learn how to trade using portfolios of algorithmic trading strategies please consider joining <a href="http://newsite.asirikuy.com/">Asirikuy.com</a>, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading.</p>