This is the first of what will be a series of posts about my attempts to develop a trading system using an evolutionary genetic process. I'm sharing these notes in the Wave59 forum for several reasons:
- 1) They might be useful to others out there who are trying to develop "machine learning" and "data mining" solutions to trading.
2) Some of you might have suggestions about how I can improve the development of THIS learning machine. Those suggestions might be rooted in your experience using certain trading strategies, indicators, or forecasting tools. Alternately, suggestions might come from your own experiments in machine learning.
3) We might discover ways to collaborate.
4) Some readers might simply find the notes interesting.
The learning machine that I'm creating is not the same as the one Earik has built into recent versions of Wave59. Even so, there might be areas of similarity. I specifically wanted to start the development of THIS machine without having prior knowledge of how a genetic algorithm "should" work. It will be interesting to see whether the two approaches went in the same directions. (I might have taken some wrong turns.) Anyway, now that I have brought the machine to a certain point, I no longer feel a need to work on it in isolation.
Overview of the prototype machine
Price data and pre-calculated indicator values are housed in Microsoft Access database tables – a data repository – which currently includes information from the period of January 2010 to October 2015.
The evolution program was written in Microsoft's Visual Basic for Applications, and resulting data about the evolution of trading strategies are housed in a separate database that links to the tables in the data repository. Theoretically, this would allow for multiple versions of the evolution program to work simultaneously while drawing from the same data repository, but right now I only have enough computing resources to allow for one evolution program to run at a time.
The programming and the data architecture were constructed with the following objectives:
- * Discover trading strategies that work intraday, without allowing positions to be held overnight.
* Initially work with ES price data, but be adaptable to other markets.
* Standardize all trading indicators in a way that allows them to be subjected to the same types of evaluation. This approach allows each indicator to be replaced by any other eligible indicator as dictated by the fitness tests and the randomized events of the evolutionary process.
* To reduce computation time, calculate all indicator values before the evolutionary process begins. These values are stored in the data repository, along with actual price data. Nearly all of the indicator values were originally calculated in Wave59 and then exported. There are advantages and disadvantages to this approach.
* To allow for multiple perspectives on the market environment, use indicators from multiple time frames simultaneously. This might be the largest area of difference between this machine and Wave59's genetic algorithm.
The primordial soup - base data and indicators
In addition to standard OHLC values for price bars, most of the indicators used in the prototype were created so they can be calculated in the same way for different time frames. The time frames used in this intraday system include 60-minute, 15-minute, 4-minute and 1-minute bars, each stored in separate tables. The evolutionary process can "select" specific indicators to include in a trading strategy from among the entire set of indicators. Thus, indicators and their values are the components from which a trading strategy is built.
The choice of which indicators to include in the data set must balance the desirability of having more permutations of an indicator to represent market conditions, versus having fewer permutations in order to make the computation process efficient. Even without considering the calculation settings for each indicator, the use of multiple time frames already introduces multiple permutations. On the other hand, having only four time frames constrains the solution space in a way that I hope makes the problem more manageable.
The guiding principle I have used in selecting indicators is that each one must provide a perspective on market conditions that is substantially different from what other indicators in the system provide.
The aspects of market conditions that I can think to capture (so far) include:
Momentum, strength of trend, direction, etc. – These are expressed with various derivatives of the Adaptive CCI. An advantage of using the Adaptive CCI is that its output is normalized with respect to its zero line. Values rarely exceed ± 300.
Cycles, periodicity – This is described with a version of the Lomb periodogram.
Wave structure (1) – For this, I am using the Wave59 9-5 indicator, with special attention to "terminal" values of 8*, 9, 14* and 15.
Wave structure (2) – the system includes indicators describing the number of consecutive higher highs, lower highs, etc. that have preceded the bar that is being tested.
Market impetus, energy, etc. – Currently, I use various calculations based on the intraday NYSE TICK Index to stand in for the force that's behind a price move. I would like to include more indicators that are based on the new Wave59 Energy Bars and the bid-ask difference, but my data set for those things only goes back to the beginning of 2015. (This suggests an area of future exploration – acquire historical tick-by-tick data and translate it into Wave59 format.)
Specific events – An event, such as price poking through the Bollinger band, can sometimes foretell a change in trend. However, the event usually occurs earlier than the ideal trade entry time. To codify events such as this, the system notes the specific price bar on which the event occurs, and then it begins a decay count that goes from zero (the event bar) upward. Thus, the indicator value that gets incorporated into a trading strategy might be a decay value that lies within a certain range. For example, a trading strategy might look for an upper Bollinger band poke that is no more than four bars old. Other types of events include various momentum divergences, and they also work with decay times.
Data series for all of the indicators described above are included in the system for each of the four time frames. However, not all of the data series are "turned on" for a given experimental run. I have tentatively concluded that there is some benefit in choosing which indicators to add during successive generations of the evolution to guide and sculpt the population of trading strategies.
Inexplicable stuff – The system allows incorporation of esoteric functions, such as astrology calculations, as long as they can be expressed as having output that is inside or outside of specific ranges of values. I don't know much about astro trading, but Earik's Buttonwood thread inspired me to include the progressive and spherical indices in the current run of the evolution program. I'm sure there is more that could go here. Astrology represents the only type of indicator currently in the system that is completely independent from actual price data, making it (hopefully) predictive rather than derivative.
Other things – The system includes a small number of odd data series that don't fit into the above categories, including time of day, NYSE Net Issues, and a few others.
Market data that is not included -- One important aspect of market conditions that is not included in the system is chart geometry. As Earik mentioned in the Buttonwood thread, it is difficult to tell a computer how to interpret lines and curves in relation to price bars. I don't think it's impossible, but my earlier attempts at working on this problem didn't make much headway. As a framework for handling this kind of thing in some future version, maybe it would make sense to describe geometric objects on the chart (e.g., a line) as a field radiating out from the object. The system could evaluate the proximity of the highs and lows of price bars (or some other way of measuring price) in relation to that field. Perhaps the fields from different objects would interact. (I'm just speculating here.) This does not seem like the most urgent direction for future research.
In the next post, I'll describe how the candidate trading strategies are constructed.
Happy new year, everybody!
! I wanted something more customize-able, and also something that one wouldn't have to pay an "arm and a leg" for. I have a working Genetic algorithm and Neural Net, and putting the two together, an Evolved Neural Net. Additionally, I have a good framework to incorporate trading with the AI.
(Incredible!). I did read a lot on the web to try to find those improvements. On another note, I am starting with the basics of AI, verifying its foundation, then to more advanced AI methods where I could get away from Curve Fitting.