Systems Workshop: "Buttonwood" E-mini daytrading system

[Simon.B]

Thanks Earik,

Really great thread. Wanted to ask a few questions, I assume that these are all on 5 min chart, and that it's
regular market hours chart only ?

Another thing - I see some 35% and larger draw-downs, I did not expect to see such large ones on such short time frame
system like 5 min, is that within normal range ?

Thanks,
Simon.

[earik]

Hi Simon,

Thanks! Yes, this is all regular market hours only. Since we're using an 8/34 crossover as our triggers, we'd get killed trying to put something like that on in the night session. Keep in mind the stage that we're working at here - basically trying to nail down what the market weather looks like so we know when to trade, and when not to trade. To do that, we're using the crossovers, just because we need something there, but I really don't think it will be hard to improve on those triggers when it's time, so don't take any of the numbers too seriously except in relative comparison to each other.

The best measure of adverse movement is the max drawdown number, which is at -$4k with the last system we looked at. I'm guessing in the end we'll get that down to probably in the -$3k range somewhere, which is a risk number I'd be very happy with. Percentage-wise, depends on what the actual account size is.

Regards,

Earik

[Feinberg]

Earik,

There is a script that you wrote (May 6, 2005) called Detrend Price Oscillator.

Is this something you (we) would want to include in Buttonwood?

Joseph

[Simon.B]

Thanks Earik, I only paid attention to the percent number...of course that is based on account size which at this point is not
important, thanks for reminder

It is really great to watch your work and logic which is used in progress. Especially because for me single biggest struggle
has always been scripting, as you know. I am sure everyone appreciates this thread....thanks again for starting it.

Simon.

[supracharger]

Thanks Earik. It's good to see the process of how you create a mechanical trading system.

- Andrew

[earik]

Hi Joseph,

Yes, that would be one to look at for improving the trigger function (Buttonwood_Signals), which right now uses moving averages. I strongly suspect there are better solutions than that. I'll leave it to you to do the testing on that one though - basically need to rewrite the Buttonwood_Signals function, and the system will work using the new entries/exits based on whatever you put in there. Probably should wait until we have the weather forecasting part a little closer to being done, but sometimes it's good to know what your homework is going to be.

Earik

[earik]

Down our first rabbit hole!

I think one of the most important characteristics of a good system developer is curiosity. That's because there are so many different rocks that we stumble onto when we are building systems, and you really have to be willing to put forth the energy and start looking under them, for no other reason than to just see what is there. In the case of this thread, everything I've bolded in red font represents a potential improvement to the core method. One person just doesn't have the time and energy to look at all of them, which is why I don't go through each one, but if I see a tempting rock, I'm still willing to take a peek. So that's what I did with our Spherical function, and whoops, I found a rabbit hole under that rock! You can officially consider this thread having gone of the rails, as this is all new territory for me, and it looks like we're going to end up in a different place than my original intention with this script...

Anyway, I got to thinking about the aspects in the spherical function we build in Buttonwood_v2, and thought I should take a closer look at them. So what I did was run the optimizer over everything from 0 to 180 in steps of 1 degree. Technically, these aren't degrees, since the aspects are actually measured as ratios of the distance to the planets compared to the diameter of the celestial sphere, which is completely different. But to keep it in terms that everyone can easily understand and relate to (including myself), I transform those values into the range of 0 to 180, so it all works out like regular old planetary aspects. But just so you know, this is much more complicated than what the other astrologers are working with. Going from 2D to 3D is a big deal, and there's a reason why Gann had two full time professional astrologers on the payroll, figuring all this stuff out and doing calculations for him.

Once the optimizer was done, we can build a chart in Excel to show the results in a visual way, which is easier to look at than before. This is the result of running Buttonwood_v2, but with the filter only building the weather index one degree at a time. So if we set the degree to "30", it will only trade on days when there are lots of 30 degree hits from the current planets to the natal, measured using spherical astro.

Average Trade by Degree

avg_trade_spherical.png (13.7 KiB) Viewed 28112 times

This is what that looks like. The vertical axis is average trade value, and the horizontal axis is the degree value used in our spherical filter. This shows what our system looked like when run on every possible degree selection in the market weather forecast. In order to make this a little easier to look at, I replaced the value at degree=0 with $30, because it was $375 and ended up squashing everyone else. You can make out some peaks and troughs, but it's noisy. Let's smooth this out, by averaging each point with its nearest neighbors. So the new value for point 30 is actually (point 29 + point 30 + point 31)/3. Do it once, then do it again to make it even smoother, and this is what you get:

A little smoother, please

avg_trade_spherical2.png (11.02 KiB) Viewed 28112 times

Just to make sure everyone is on the same page here, let me take a step back and take a minute to explain again what we're looking at. Basically, we are trying to find big days in the ES, because those days are the easiest to trade. So what we did was build a trend-following system, and we are only running it at selected times, according to the "weather" filters we are working on. We then study the average trade value from the system reports. When the average trade value is high, it means that the system is doing well, and making lots of money. When the average trade value is low, it means the system is struggling, and not making as much money. So the average trade value is a representation of market volatility. We could try and measure market volatility directly, and there are ways to do that, but we are more interested in making money than forecasting volatility, so it actually makes a lot more sense to measure volatility as the success of a trend following system instead. So that's why we like high average trade values, and are using that in order to build our weather functions. If we can figure out what days moving averages work well, then we'll be able to make money using all the other trend following technical indicators as well, because if moving averages work, then they will too. Make sense?

Coming back to this last chart, you can see that the average trade value does really well towards zero. It then cycles out, losing performance as we get towards 33 degrees, then gaining it back towards 53, etc. You'll notice that I'm writing this post on a Saturday. That's because I got all excited about that chart last night, and saw a cycle there. Check it out:

Secrets of the Universe, or Shapes in the Clouds?

looking_for_cycles1.jpg (144.46 KiB) Viewed 28112 times

A little messy, but it's a worksheet. Remember those IQ tests they used to make you take back in high school? The solution (top line) reminded me of that. Basically, start off with a small cycle, then each new wave adds a fixed amount. So at point 3 at the top, we're at 24 degrees. The next point is 10 degrees forward to a low at 34, then increment by 2, which gives us 10+2=12, and add 12 to 34 to get the next high point at 49. We incremented by 12 last time, so add 2 again to get 14, and then move forward that amount from 49, to 63, which is the next low point. As we move forward, we increment each additional time, making swings that are larger and larger. The increment value then starts decreasing, from +2, to +1, then to +0.5. So it's all nice and regular, and the final swing wraps around to 184 degrees, which reduces back to 4, giving us the peak at the first part of the chart. Anyway, if it's not totally clear, that's ok. Just showing what I was working on here - a very regular, very predictable cycle that describes the ebb and flow in the volatility cycle we've been looking at. Cool stuff.

Everything in our galaxy is thought to orbit around a big black hole, at the very center. That black hole is so big and strong that entire galaxies orbit around it, just like the Earth orbits around the Sun. As you get closer to the center of the black hole, the gravitational force gets stronger, so the stars located there have to move faster in order to avoid getting sucked into the hole. That's actually how astronomers figured out there was a black hole there - the nearest stars were moving at crazy fast speeds, flying around it. Then, as you move out, everything slows down. It's the same in our solar system - Mercury goes around the Sun faster the Venus, which goes faster than the Earth, etc.

Anyway, that's what this reminds me of. Sort of a Kepler's Laws of Planetary Motion, but for markets. The highest energy is located at 0 degrees, which is a spherical conjunction. You know it's the most powerful aspect because you can see the huge spike in average trade value there. Then, as we move out, we do so in waves, reducing our average trade, then increasing it, etc. The farther we get from zero, the slower the waves move, and the closer we get to zero, the faster they move. So here we are, messing around with filters and mechanical systems, and the next thing you know we're talking about secrets of the Universe. That's Wave59 for you. LOL

Let me also just take a moment and point out that there at 10 points here, starting with a high just after the zero point. If you've bought the Bertha book, that's going to get you excited, as the Kepler piece has some really interesting implications to those techniques. If you haven't read the Bertha book, just ignore that comment and don't worry about it. If you have read the Bertha book, but didn't actually buy it, well, then no cookie for you either way - Santa's going to put a lump of coal in your stocking this year.

So you can see why I was up messing around with this last night, and why I was excited enough to get in here on Saturday and do some more work on this. It's interesting stuff, and forecasting volatility is no small trick. If we could forecast daily volatility accurately, as far ahead of time as we wanted, we'd basically be able to print money in the options market.

However, now that we're here, it's good to take a step back and remember what we are doing, and that we've been running down a rabbit hole. We're in the middle of building a day trading system, and need to make some decisions. What are we going to do with this information as far as our daytrading system goes?

As much as I like the pattern that I stumbled on to, and as much as I'm drawn to solving markets in a more large scale theoretical way, there are some issues. First, there's a little bit of a "curve fitting" feeling I've got, working all of this out. Yeah, it makes theoretical sense, and it's very cool, but we used the 2009-2015 data to solve for this volatility cycle, so if we implement it in a very complicated and detailed way, it will obviously do really well from 2009-2015, but how will it do in 2016? Do I really trust all those minor cycle highs and lows towards 180? It's the complexity that has got me a little worried. Part of building systems is data mining, and there's no way around that, but there's a point where you've got to realize that the model is getting a little too involved, and I'm there with this particular approach. It might be the holy grail, but I'm not trying to solve for that right now - I'm just trying to build a reliable system. So I'm not going to forget that I saw this, and will come back to it later, but I'm going to see if maybe there's a way to just simplify a little and consider robustness over perfection.

In other words, we've got this data, so what's the most obvious, in-your-face sort of thing it is telling us? Yes, the cycle is there, but even more blatant than that is that the average trade value is really skewed towards zero. What if we focus on that? In other words, rather than worrying about how all those stars orbit around the black hole, why don't we just focus on the black hole itself. Let's go to where the odds are completely in our favor.

That means building a revised spherical filter, which I'll do next. Climbing back out of the rabbit hole....

Earik

[earik]

Buttonwood_v3: New Spherical Filter

Following up on the last post, I've decided to build a new spherical filter that will only focus on the conjunctions, which seems to be the most significant area of the plots shown last time. Note that I'm ditching the 90 degree aspects, as that aspect really isn't working for me anymore after taking a look at the full 0-180 degree charts. Yes, there's a blip there, but I want a big mountain rather than a blip.

Now that I'm clear on what the filter looks like, I'm also going to try and nail down some of the settings a little tighter as well. In particular, the correction setting is bothering me, since we've been using a rough estimate. So what I did was add the Ascendant (degree of the eastern horizon, abbreviated as "ASC"), as well as the Midheaven (degree straight up, abbreviated as "MC") to the spherical filter, as they weren't in there before. Then, I commented out various parts of the function so that we only got hits when current planets passed over the natal Moon, natal ASC, and natal MC. The Moon moves 15 degrees in a day, and the ASC and MC move 360 degrees in a day. If we only get hits on those particular points, that will give us an indication what the correct natal time is, because if we're off by a few minutes, the ASC/MC will be in the wrong place and the results won't be as good. So basically, if we only look at hits on the fast movers, we can use that to tell us when our times are correct and incorrect.

In traditional astrology, this is known as chart rectification - you basically work backwards, using events in the person's life to solve for their birth time. So if the person broke their leg, you'd look for a bad aspect, probably involving Mars. If the progressions you use make an exact hit on the day they broke their leg if the persons time of birth was 9am, but they don't work if the birth was 10am, then you concude that the correct birth time is closer to 9am, since that's what is required in order for the techniques to be accurate. The assumption is obviously that the techniques are accurate to start with, which then lets you reverse engineer the actual birthday, working backwards from known events.

We can do the same thing. We've got evidence that conjunctions from current to natal planets using spherical astrology have a high correlation to large range days in the ES. We can then use that technique on a range of starting points, and see which ones give us the best results, assuming that the best results happen when we use starting times close to the actual ones. Since we've removed the slowest planets from the mix, it will make it easy to see which times are "hot" and which are not, since we need to be exact with time in order for the Moon/MC/ASC to perform.

I ran a really big optimization from one day before 5/17/1792 to one day after, looking at 20 second increments. We know the day, we just don't know the time, so that seemed like the best way to solve for it. I don't have a graphic for this one, but can say that the correction value that jumped out at me (surrounded by profitable nearby neighbors) was 0.5236. If that's right, it means Jack Gillen was half a day too early, and rather than 9:00 in the morning, the birthday would be much closer to 9:00 at night.

Re-building the spherical filter lets us do a check on the threshold values, to see what they look like now:

Testing a range of thresholds on the new filter

threshold_testing_on_new_spherical_filter.png (38.43 KiB) Viewed 28112 times

Do you see what happens with the average trade value? As we require higher thresholds, the value goes up. That means that as we get more hits on the index, the market volatility gets larger and larger. That's exactly what we want to see.

So the settings for the new spherical filter are:
orb = 8
correction = 0.5236
thresh = 6 or 7

The 0.5236 correction is going to be a little different than what we were using before, so I wanted to state it explicitly since this thread is becoming pretty long. Note that the correction value is different from the 0.25 that worked best with ProgressedToNatal, which is what we were using before.

Here's what the equity curve looks like (threshold = 6):

Equity Curve for new filter

new_spherical_equity.png (45.89 KiB) Viewed 28112 times

So to summarize where we're at, we have two good filters right now:

1) Progressed-to-Natal, average trade $28.19 (at correction = 0.25, orb = 1)
2) Spherical-Conjunctions, average trade $29.40 (at correction = 0.5236, orb = 8)

The other ones aren't making the cut, so unless someone figures out how to get them to give us average trades of at least $25, I'm going to stop paying attention to them. I notice that each of our filters requires a different correction setting, and both drop if they are moved off of their preferred setting. ProressedToNatal drops to $22 if using spherical settings, and spherical drops to $27 if using progressed-to-natal settings. Not sure if I'm OK with giving them different natal dates or not just yet, but for now that's fine, since they're both still profitable even if they get moved off their ideal settings, which is what we want to see.

That's a lot of testing on weather functions. <whew!> We might be ready to look at finalizing this first part of the system soon, as I think we've got something that will give us the edge we're looking for.

All for now!

Earik

[orionsbelt]

wow! great stuff, Earik. here are some possible suggestions for everyone to ponder that i have been thinking about.
what about specific days of the week. are certain days better trending days on average than others?
what about the days before or after holidays or holiday weekends? do we keep them or omit them?
what about specific times of the day? should we only trade in the mornings? should we trade or skip the first 30 or 60 minutes after the open? trade/skip lunch time?
what about days following high or low volatility days?
what about days following high range or low range days?
what about gaps? is it better to trade days with larger or smaller opening gaps?
are the days better to trade following a range bound or a trending pre-market?
what about the initial or opening volume (the first 1 or 5 or 15 minutes?) of the ES emini? does above or below the average volume make a difference in the kind of day that may develop?
can we use how the bond market opens up to our advantage?
how about the overseas markets? large trending day or not?

i don't know if this will help or not in finding better market weather situations to trade or not and i don't know how to test for them very well but i figured i would throw them out there.

thanks again

[earik]

Hi Orionsbelt,

Thanks! These are all great suggestions. In the next version, I'll add another filter area so people can start to test some of these things out. My goal is to try and make it easy enough so that people with limited QScript experience can still feel able to jump in there and try testing some of the easier things. I'm not sure if I'll succeed, but that's the goal at least!

I built an intraday system about a year and a half ago, and went through a very similar brainstorming process to get a list just like the one you posted. You can learn a ton about markets by testing those sort of things out, and it can be the difference between a good system and a great one. I'm not willing to test 20 filters myself and post all the system reports here, because that's just a TON of work, but if the pile got split up between everyone, it could go very quickly and we'll end up with a much better system in the end because of it.

Anyway, watch for this in v4, as pretty soon we need to look at this stuff since we'll have the primary weather forecasting piece completed.

Regards,

Earik