Home

Products
Data Factory FAQ

Click to EnlargeCQG Data Factory™ offers traders more than twenty years of End of Day market data and more than seven years of intraday data, including Time & Sales (tick data), intraday bar, and trade volume. Customers can also access additional data from as far back as the 1930s. Some of our most frequently asked questions concern:


 
Format of Data Factory Data
The CQG Data Factory can produce files in a variety of time intervals:
  • Time & Sales Data (Tick Data)
  • Tick Volume 
  • Intraday Bar Data (1-60 Minute Bars)
  • Daily Bar Data
  • Daily Volume / Open Interest  

Data is delivered for a CQG trading day that typically runs from 16:00 one day to 15:59 the next day (all time stamps are U.S. Central Time). This is not the same as a calendar day. The evening session will be included in the next trading day data. Sessions are set up in accordance with exchange specifications. The exception is that Data Factory data requires an additional session for sessions spanning 00:00 CT. Sessions may not be consistent across contracts, but tend to be consistent for a specific contract over time. The data is delivered with CQG PC symbols. These symbols might correspond to exchange symbols.

Data File Names
The contract name is in the format F.US.PPSSMYY, where:

  • F is the type (F for future).
  • US is the country code (US for all futures).
  • PS is the CQG PC Symbol for a commodity (indicated by one or more characters).
  • M is the contract month.
  • YY are the last two digits of the year.

For cash contracts, the contract name is formatted as X.CC.PPSS, where:

  • X is the type (X for cash or index).
  • CC is the country code.
  • PPSS is the CQG PC Symbol (identified by four or more characters).

The type of data in each file is identified by a file name extension. The CQG contract name appears before the file name extension.

Data Type    Extension
(T)ime & (S)ales   .ts
(B)ar Data - (03) Minute Bars  .b03
(B)ar Data - (15) Minute Bars  .b15
(B)ar Data - (45) Minute Bars  .b45
(B)ar Data - (60) Minute Bars  .b60
(B)ar Data - (D)aily Bars  .bd
(V)olume -   (D)aily Volume  .bd

Examples:
F.US.US92M.B03 (F=Future, US=United States, US=Treasury Bonds (Pit), 92=1992, M=June, B=Bar, 03=3 minute)
X.US.SPC5.BD     (SPC5=S&P 500 Cash, B=Bar, D=Daily)

File Format of Time & Sales, Bar, and Daily Data
Time & Sales (Tick Data)
Each record contains eight data fields separated by one or more spaces. The fields are:

CQG Contract Name   6-14 characters
Trade Date    8 digits - yyyymmdd
(S)ession Indicator   1 character - 0=First Session, 1=Second Session, 2=Third Session
Transaction Time   4 digits - hhmm (U.S. Central Time)
Transaction Price   1-7 digits
Transaction (T)ype   1 character - B=Bid   A=Ask   T=Trade   S=Settle
Transaction (M)arket Indicator 1 character - F=Fast Market     N=Normal Market
Transaction (C)orrection  Indicator 1 character - E=Exchange Delete I=Insert F=CQG Filter Delete   N=Normal

Note: For an in-depth explanation of time stamps and session indicators for Time & Sales files, view the Data Factory in Depth document.

Example:
Contract    Date     (S) Time Price (T) (M) (C)
F.US.SPAZ02 20021101  0  1545 88200  T   N   N
F.US.SPAZ02 20021101  0  1545 88200  T   N   N
F.US.SPAZ02 20021101  0  1545 88170  B   N   N
F.US.SPAZ02 20021101  0  1545 88200  A   N   N

Tick Volume Extract
Each record contains nine data files separated by a space between columns. The fields are: Contract Name, Date, Settle, Time, Price, Trade, Transaction Market Indicator, Transaction Correction Indicator, and Volume.

Example:
Contract    Date    (S) Time Price (T) (M) (C) (V)
F.US.JNKH08 20071126 0  0115 15210  A   N   N   10
F.US.JNKH08 20071126 0  0115 15210  A   N   N   15
F.US.JNKH08 20071126 0  0116 15210  B   N   N   1
F.US.JNKH08 20071126 0  0116 15210  A   N   N   15
F.US.JNKH08 20071126 0  0117 15210  B   N   N   1
F.US.JNKH08 20071126 0  0117 15210  A   N   N   17

Intraday Bar Data
Each Intraday record contains eight data files separated by one space between columns. The fields are: Contract Name, Date, Time, Open, High, Low, Last, and #_Price_Changes.

Example:
Contract    Date     Time  Open   High   Low    Last   #_Price_Chg
(not trade vol)
F.US.DDZ02  20021101 0205  31700  31700  31600  31600   15
F.US.DDZ02  20021101 0206  31600  31630  31585  31600   21
F.US.DDZ02  20021101 0207  31590  31660  31590  31620   17
F.US.DDZ02  20021101 0208  31620  31660  31605  31605   13

Daily Bar Data
Each daily records contains six data fields separated by one or more spaces. The fields are: Contract Name, Date, Open, High, Low, and Last.

Example:
Contract    Date      Open    High    Low    Last
F.US.DDM03  20021101  31400   32250   31400  32230
F.US.DDM03  20021104  33935   33935   33935  33935

Volume & Open Interest (Daily) 
Each Volume & Open Interest record contains six data fields separated by one or more spaces. The fields are: Contract Name, Date, Commodity Volume, and Commodity Open Interest. Note: Commodity Volume and Commodity Open Interest are the sums of all Volume & Open Interest for all maturities of the commodity.

Example:
Contract   Date  Volume Open Int. Volume Open Interest
F.US.QEAF02 20021101 0  1         589237 2176447
F.US.QEAF03 20021104 0  1         589237 2176447

Data Factory Addendum for End of Day File Format
The End of Day File format is similar to the Time & Sales format. It includes columns for Contributor Identification, Tick Flags, and Trade Volume.

Forex Sample
* CID TickF Vol
X.US.USDZAR 20051206 0 1600    63205 B N T 085 --BNN 0
X.US.USDZAR 20051206 0 1600    63455 A N T 085 --BNN 0
X.US.USDZAR 20051206 0 1606    63160 B N T 113 --BNN 0
X.US.USDZAR 20051206 0 1606    63460 A N T 113 --BNN 0

Futures Sample
* CID TickF Vol
F.US.QEAZ05  20051206 0 0737    97535 B N T 000 --NFN 30705
F.US.QEAZ05  20051206 0 0737    97540 T N T 000 S-BNN 260
F.US.QEAZ05  20051206 0 0737    97540 T N T 000 --NFN 256

Field Definitions
The "Transaction (C)orrection Indicator" has a value Normal = "N." In End of Day format, the value Normal is designated by "T."

CID: Contributor Identification indicates which contributor sent the quote. This is not used for futures, as the exchange is the only contributor. It is used for Comprehensive FX (such as USDJPY).  This could be used for any consolidated feed in the future, for example U.S. equities on CTS.

 

TickF: Tick flags indicated by five characters, XxyzN, where:

  • Xx - Sales Condition Indicator
    • -- - no sales condition
    • N - notional settle
    • ba - indicative bid/ask
    • Z - out of sequence
    • C - cash sale
    • G - bunch sold
    • N - next day
    • O - open out of sequence
    • R - seller
    • S - spread trade
    • T - formT out hours
    • W - average price
    • b - indicative bid
    • a - indicative ask
  • y - Intraday/Historical Processing Flag
    • B - both intra/hist
    • H - historical bars only
    • I - intraday bars only
    • N - neither intra/hist
  • z - Flat Filtered Flag
    • F - tick is flat filtered
    • N - tick not flat filtered
  • N - Unused, Reserved For Future Flag

Vol: Trade Volume and Bid Ask Size: This is generally available on electronic markets, but not on open outcry markets. Full trade volume is available when sent explicitly by the exchange, that is, not as cumulative volume.

CQG Data Factory Time Stamps
There are two distinct sets: time stamps for Time & Sales data and time stamps for bar data (1 minute to 60 minutes). The time stamps for Time & Sales data are by trade date, and the time stamps for all other intraday data (1-minute bars, 5-minute bars, etc.) are chronological.

All time stamps are U.S. Central Time. These are adjusted for U.S. Daylight Savings. 

Futures
A similar pattern will be observed with futures contracts such as EP and SPA, which trade overnight. However, there is one additional consideration for the fact that a session spans midnight. This is not evident in the previous example because the first session ended at 23:59.  

The sessions for EP (E-mini S&P 500 futures) are:
1545 - 0829    Globex Night
0830 - 1515    Globex Day (plus settle)

The first session spans midnight. Since the CQG database archives data by chronological date, data for this session is drawn from two different files. Therefore, Data Factory will mark session 0 as 1545 - 2359 and session 1 as 0000 - 0829. The second session, Globex Day, will be marked as session 2.

Additional Support
Use our Symbols and Exchanges search to view updated symbols lists and session information.

Two important notes:

  1. For the electronic platforms of CME and CBOT, please refer to Globex and e-CBOT, respectively.
  2. The search lists sessions beginning with session 1, 2, etc., whereas the Data Factory begins with 0, 1, etc. Recall that the Data Factory inserts an additional session at midnight for sessions that span midnight.

Saturday Time Stamps for Certain Contracts
Contracts such as SPA (CME) and DFA (CBOT) have evening sessions Sunday - Thursday that open the trade dates Monday - Friday like this (as defined by the exchange):

Day of week                         S M T W R F S
Evening Session                     X X X X X      
Day session (inc. settle)             X X X X X

Sessions of the same color belong together in one trade date. The evening session is immediately prior to the day session in each case.

Contracts such as JGB (TSE) and AP (SFE) have evening sessions Monday - Friday:

Day of week                       S M T W R F S
Evening Session                     X X X X X
Day Session (inc. settle)           X X X X X

Monday evening sessions open on the Tuesday trade date, and Friday evening sessions open on the Monday trade date. Friday evening sessions are exceptional in that there are two calendar dates (Saturday and Sunday) in between.

Sydney is 15 hours ahead of Chicago, and our systems run on U.S. Central Time. Given that Sydney contracts trade on a roughly 24-hour basis, the beginning of a session will be up to 37 hours ahead, or, in the case of the Friday session, up to 85 hours ahead. This comes up against system limitations. In order to capture the evening sessions for contracts like AP and JGB in their correct sequence (that is, evening session first, day session second, not the reverse), we have to configure our system to accept the session by pushing the session forward one trade date. This should be transparent to the user except in the case of a non-sequential session progression, such as a trade date comprised of Friday-Monday sessions. 

Data Factory must necessarily capture Friday evening sessions with a Saturday time stamp.

Data delivery by FTP
CQG offers data delivery by FTP. This allows for quicker delivery and eliminates shipping costs.

Before data can be posted to the FTP, these conditions must be met:

  • A valid order, including specific symbols, dates, and formats, must be received.
  • For new customers, a billing account must be created.
  • Billing (preferably to a credit card if you’re a non-CQG customer) must be completed satisfactorily.
  • The data extraction from the database needs to be completed, and the file must be posted to the FTP.

To access the FTP:

  1. Click ftp://ftpb.cqg.com/.
  2. Click the File menu, and then click Login As. You can also click ftp://login:password@ftp.cqg.com/.
  3. Enter your user name and password. The password is case sensitive.
  4. Click ftp://sample:CQG@ftp.cqg.com/ for Data Factory documents.

Extracted data files will be in the format login.Daily/Weekly.YYYYMMDD.order_id.part_number.zip, and will be in folders named 20021117, 20021118, etc. These folders will be removed after fifteen days. 

Downloads from the FTP must be done in binary (BIN), not ASCII format. If you are having problems logging on, make sure you are in binary mode. In addition, it is generally not possible to log on directly to a sub-directory, such as login/20021117. Finally, if you do not see your folder update, refresh the browser.

Data File Compression
The data files are compressed by CQG to approximately one-fifth to one-twentieth of their normal decompressed size. The files must be decompressed before they can be used. Compressed files are in ZIP format.

To decompress the data files, download ZIP Reader by PKWARE or WinZip.
Microsoft compression on XP® uses a different standard. If the format is not recognized on Microsoft XP, install a standard format compression, such as WinZip®.

When the data files are decompressed, they become ASCII text files on the hard drive. The ASCII text files can be loaded into standalone systems, spreadsheets, or other software packages for analysis. 

Related

Extended Historical Data
Markets Covered

 

Copyright © CQG, Inc. All rights reserved.

CQG, Inc. has registered CQG®,  DOMTrader®, SnapTrader®, TFOBV®, TFOBVO®, and TFVOL®.
TradeFlow™ and TFlow™ are trademarks of CQG, Inc.