Database Compression and Smart Tick Compression
This is a feature that can greatly reduce the size of your database.

The Database Compression feature has been enhanced to apply Smart Compression of Tick Data when compressing the Intra-day Database, if Smart Compression option is enabled.  If Smart Compression of tick data is enabled, performing a Compress Database on the Intra-day database will now compress the entire tick data collection for all symbols in one step.

**NOTE** - While smart compression will be a very good thing for most people, it can have adverse affects on historical data in ways not expected.  For instance, if using an indicator or Footprint chart filtered for certain size trades, live steaming data will create the picture you expect. HOWEVER, when smart tick compression is turned on and if you reload a chart (close, re-open, or reboot the software) the data you see now will be in a compressed format. I.E. The ticks will have larger sizes (merged/compressed) and would lead to much more "large size" trades.

During our testing, Smart Compression reduce the tick data (number of ticks) consistently by over 85%.  Users can now store approximately 7 times as much tick data in the same amount of space.  This compression has a number of very positive impacts on the performance and capability of MarketDelta: users can store more tick data, charts load and process quicker, databases verify and backup quicker, faster backtesting.  The list goes on and on.  Users that once store 3 days of tick data will now be able to store 20 days of tick data using less space in the database.

When Smart Compression is on, very few aspects of MarketDelta are adversely affected.  The one aspect that will affect most users will be tickbar charts.  If tickbar charts are a big part of your analysis, then Smart Compression might not be for you as tickbar charts will not function properly with compressed ticks.  However, ALL other periodicities (Volume Bars, Range Bars, Second Bars, etc) will be identical with Smart Compression turned on.  Also, advanced volume functionality like the Profile Indicator (Volume at Price / Delta at Price), the Volume Breakdown Indicator (Delta per Bar), and VWAP, will also be unaffected by  Smart Compression.  Certain results of the Volume Breakdown Indicator can be affected.

What aspects of MarketDelta can be adversely affected when Smart Compression is turned on?

In a nutshell, any aspect of of MarketDelta that relies knowing the actual number of raw ticks over a given period, will not work accurately with Smart Compression.  These include...

• Tickbar Charts or Periodicities
• Specific Volume Breakdown Indicator Options including:
  - VB is setup to be based on "Trades" instead of "Volume" (top right setting in Prefs window)
  - VB setup with any "Result" that is based on tick counts including:
            - Total Traders/Ticks
            - Average Size
            - Total Traders/Ticks
            - Big Trader %
            - Average Buy Size
            - Average Sell Size
            - Buy Traders/Ticks
            - Sell Traders/Ticks
  - VB setup with a volume filter "Consider Trades with Volume" above, below, or between a given level.
   
  

When Smart Compression is on, ticks are stored in the database in a compressed format by combining similar ticks which share the price, bid, ask, and timestamp.  These similar consecutive ticks are stored as one tick with a volume that represents the cumulative volume of the component ticks.  Once this option is turned on, all new tick data that flows into the database will be compressed.  This includes ticks that result from downloads/backfills and live ticks that are flushed to the database at the end of each session and each time the user quits the program.

Below is an example of a block of ticks that are displayed in both compressed and uncompressed (raw tick) formats.  Notice the first tick matches, then the price changes and the next 3 uncompressed ticks became on one compressed tick.  Then the bid and offer changed and the next 22 raw ticks became one compressed tick with a cumulative volume of 220 (sum of the size of the 22 raw ticks).

Intraday Disk Space Requirements

1-Minute, Compressed Ticks, and Uncompressed Ticks

The MarketDelta database is limited to storing 2gb of intraday data (combined minute data and tick data on all symbols).  A recent analysis of data on the ES revealed that, on average, one day of 1-minute data required 0.072mb of database space, a day of compressed tick data required 0.83mb, and a day of uncompressed tick data required 6.17mb.  Based on these numbers, the MarketDelta database is capable of storing 27,397 days of 1-minute ES data, 2,410 days of compressed tick data on ES, or 324 days of uncompressed tick data on ES.  As a specific example, if you were retaining uncompressed tick data on 6 symbols with comparable volume to ES, you should limit your tick retention under 50 days.  However, compressed ticks would allow that limit to be raised to close to 400 days.