ninjatrader/utilities/tick-data-conversion/README.md

# Poor Man's Guide to Historical Tick Data for NinjaTrader

This guide covers how to import historical tick data from Sierra Chart into NinjaTrader for cheap.

A 1-month subscription to Sierra Chart (and the Denali Exchange Data Feed), which is sufficient for the purpose of this guide, is ~$40.

Services such as Portara CQG cost ~$800 per instrument (e.g., S&P 500 E-Mini Futures).

Depending on how much data you require, this could save you thousands of dollars.

## Motivation

Historical tick data is useful when backtesting strategies that depend on intrabar price movements (i.e., when OHLC alone is insufficient).

However, data providers provide only a limited amount of historical tick data.

For example, the default NinjaTrader Continuum data feed provides only a year's worth. IQFeed, my current data provider, provides up to 180 days only.

In order to obtain more data, it must typically be purchased from a data vendor, but as we've seen, this can be very expensive.

Hence the desire to obtain historical data for a fraction of the cost.

This is where the Historical Data Service provided by Sierra Chart comes in. According to their [documentation](https://www.sierrachart.com/index.php?page=doc/SierraChartHistoricalData.php#Included):

> Tick by tick data for futures contracts on the CME/NYMEX/COMEX/CBOT begins at 2011.
> 
> Prior to this the CME/NYMEX/COMEX/CBOT data is in 1 minute units and begins at June 2008. 

In order to access this data, you just need to subscribe to a Sierra Chart service package.

I'm using ```Integrated Standard Sierra Chart (Service Package 10)```, but I believe any service package will do.

## Exporting Data from Sierra Chart

1. Go to Global Settings > Data / Trade Service Settings.
2. Make sure Intraday Data Storage Time Unit is set to 1 Tick and that the Maximum Historical Intraday Days to Download for 1-Tick Data is set to a period long enough to download all the data in the contract (doesn't have to be 730 days as shown in the screenshot, ~100 days should be enough).
3. Apply All changes.

![Data / Trade Service Settings](./images/data_trade_service_settings.png)

4. Connect to the data feed if not already (File > Connect to Data Feed).
5. Go to File > Find Symbol.
6. Enter the symbol for the contract you want to download data for.

![Find Symbol](./images/find_symbol.png)

7. Open Intraday Chart, then Close. 
8. Wait for historical data to download / chart to finish loading.
9. Disconnect from the data feed (File > Disconnect).
10. Go to Edit > Export to Intraday Text File.

![Export to Intraday Text File](./images/export_to_intraday_text_file.png)

11. Click save and wait on the export to complete.
12. Repeat this process for as many contracts as you desire.

## Converting Data to NinjaTrader's Expected Format

The exported tick data can be converted into the format that NinjaTrader expects by running the script ```convert_tick_data.py```.

The script depends on ```pandas```, so make sure the package is installed.

Make sure to specify the file exported from Sierra Chart as the input and name the output file accordingly.

The output file name matters, so refer to [NinjaTrader's documentation](https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?importing.htm) on importing tick data for more information.

Important bit:

> When using the NinjaTrader format, the name of the text file to be imported must be the NinjaTrader instrument name followed by a period and "Last", "Bid", or "Ask" depending on the data type. For example:
> 
> MSFT.Last.txt for Microsoft stock last price data
> 
> ES 12-09.Bid.txt for the S&P E-mini December contract bid price data
> 
> EURUSD.Ask.txt for the Euro/U.S. dollar currency pair ask price data

The script outputs the **Last** price.

## Importing Data into NinjaTrader

1. In the NinjaTrader control panel window go to Tools > Import > Historical Data.
2. Select the desired Format (e.g., NinjaTrader (end of bar timestamps)), Data Type (e.g., Last), and Time Zone of Imported Data (e.g., UTC).
3. Select "Generate 'Minute' bars from imported tick data" if you wish to do so (recommended).
4. Click Import and select the output file(s) from the tick data conversion script.

![Historical Data Import](./images/historical_data_import.png)

5. Finally, wait for the import to complete.

You will now be able to perform backtests (e.g., via the Strategy Analyzer) using the imported historical tick data.

---