Tickety, Tickety Tick

TickerTo develop and back-test trading algos its essential to have great gobs of historical data on hand, almost always at the tick level.  More often than not this data comes from your broker—OANDAAlpariDukasCopy, and Integral (TrueFX), to name a popular few—whether through commercial platforms like MT4 or proprietary APIs.  Alternately, one can use a generic downloader like Birt’s Tick Data Suite.

Over the years I’ve worked with more than a dozen brokers and other data providers so you think I’d have a handle on this stuff by now, but nothing could be further from the truth.  No, its an ever changing and messy landscape.   Continue reading

Cancel That

Cancellation in a multi-threaded Console App can be surprisingly difficult, even though I’ve done it dozens (hundreds?!?) of times before.  As any programmer can tell you, there are the “Things you know” and “Things you merely think you know.”  In programming, fiddly little details count.

I won’t go into the trade-offs between using Tasks or Parallel.For[Each] or TPL DataFlow, but suffice it to say that the later is both the most useful and easiest to get right.  Enjoy…

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;

namespace CancelDemo
{
   class Program
   {
      static void Main(string[] args)
      {
         Console.WriteLine("Press any key to cancel...");
         Console.WriteLine();

         var cts = new CancellationTokenSource();

         var client = new HttpClient();

         try
         {
            var fetcher = new ActionBlock<string>(
               async url =>
               {
                  await client.GetAsync(url, cts.Token);

                  Console.WriteLine($"FETCHED {url}");
               },
               new ExecutionDataflowBlockOptions()
               {
                  CancellationToken = cts.Token,
                  MaxDegreeOfParallelism = Environment.ProcessorCount
               });

            urls.ForEach(url => fetcher.Post(url));

            fetcher.Complete();

            var readKey = new TaskFactory(cts.Token).StartNew(() =>
            {
               while (!Console.KeyAvailable)
                  Thread.Sleep(100);

               cts.Cancel();

               Console.ReadKey(true);
            });

            Task.WaitAny(new Task[] { fetcher.Completion, readKey });
         }
         catch (OperationCanceledException)
         {
         }
         catch (Exception error)
         {
            cts.Cancel();

            Console.WriteLine("Error: " + error.Message);
         }

         Console.WriteLine();
         Console.Write("Press any key to terminate...");

         Console.ReadKey(true);
      }

      private static List<string> urls = new List<string>()
      {
         "https://www.pinterest.com/pin/143270831869696811/",
         "https://www.pinterest.com/pin/155585362100632165/",
         "https://www.pinterest.com/pin/23010648069511614/",
         "https://www.pinterest.com/pin/303711568591359002/",
         "https://www.pinterest.com/pin/314900198915723688/",
         "https://www.pinterest.com/pin/316166836311260373/",
         "https://www.pinterest.com/pin/535576580657166380/",
         "https://www.pinterest.com/pin/56154326576327972/",
         "https://www.pinterest.com/pin/56154326576368600/",
         "https://www.pinterest.com/pin/88523948898601589/",
         "https://www.pinterest.com/pin/108438303498183811/",
         "https://www.pinterest.com/pin/127156389453876236/",
         "https://www.pinterest.com/pin/270708627574674608/",
         "https://www.pinterest.com/pin/368380444492800731/",
         "https://www.pinterest.com/pin/250653535491278041/",
         "https://www.pinterest.com/pin/565483296932119884/",
         "https://www.pinterest.com/pin/173177548141309276/",
         "https://www.pinterest.com/pin/430867889333901210/",
         "https://www.pinterest.com/pin/61361613645883874/",
         "https://www.pinterest.com/pin/16466354858074206/"
      };
   }
}


Don’t Renko; Wicko!

WickoViewerThere are countless ways to chart price movement (interval or tick-count-based candlesticks, range-bars, Kagi, etc.) but my hands down favorite is Renko. Well, not Renko, per-se, but rather a Renko variant I like to call “Wicko.”  Renkos track fixed price movements irrespective of time, which is super useful if you want to filter out noise.  Unfortunately, a traditional Renko only includes Open and Close prices; not High’s and Low’s.

As to why I bothered to create my own Wicko feed (or to use the QuantConnect term: consolidator), I couldn’t find a single bug-free C# implementation even though I spent something like four hours looking.  There are plenty of contenders, of course—the internet is rife, after all, with iffy code!—but each and every implementation had one or more obvious bugs.  The rounding errors and lack of edge-case handling were the least of it, but the deal-breaker was the marked ignorance of gap handling.

Dare I say, my own implementation handles things beautifully, proved out by a bunch of unit tests.  You can download the source from my GitHub repository.

BTW, the name Renko come from the Japanese term for bricks: renga.  I’ve always liked the notion of laying down bricks on a chart, so that’s another reason to love ’em.

San Diego, Seattle, then Seattle; Oh My!!

TractorTis the season for peregrination, la de dah de dah, de dah, de dah.  For a start I’m off to a fun-filled family reunion in sweltering San Diego (in Ramona, CA, if I’m being particular; to get that gen-u-ine desert during prolonged drought experience!) followed by two back-to-back weeklong internal Microsoft conferences in Seattle.  Yes, the joys of professional development know no bounds, but if that wasn’t enough, I also get to attend a three day team offsite in Baltimore a couple of days after I get back home to Philly.

Anyway, I’m complaining overmuch because for a start (a) I love my family and (b) I’ve heard mention that this time around I get to ride my brother-in-law’s newish 1937 tractor on his “gentleman-farmer” farm, whooping it up and bellowing  my trademark “Yowza” at the top of my lungs as the barely motile beast tops an eye-watering 10mph.  To be sure, life doesn’t get any better!

Continue reading

Handling Poison; Messages, I Mean

A poison message, for the uninformed, is a “queued” message that can’t be processed for one technical reason or another.  Doesn’t sound like much of a bother, but due to the way that queues operate, they can muck up the works if left unhandled.

Azure WebJobs does a decent job of handling poison messages out of the box. Basically, you’re given five tries (by default) to process a given message.  If, for instance, the database you’re trying to write to is unreachable or maybe the instructions embodied in an message were somehow malformed then the message would be automagically moved to a “poison” queue for further processing; but only after five failed attempts.  As to any further handling, it’d be up to you.

Continue reading

I Need a [Web]Jobs

Once again, with the new formed buds of spring a-budding, it only natural for a young man’s thoughts turn to . . . WebJobs.  OK, probably not.  But this (not quite young!) man has a good reason to be thinking of such things: I’m giving a WebJobs presentation at Philly Code Camp 2016.1 later today.  The session will be more than a little hands-on, but I also have a smallish deck, with a number of highlights and resource links, that you can download: Batch Processing with Azure WebJobs.

For the uninitiated, WebJobs can be thought of the newish Azure tech that (most usefully!) enables workflow for the modern web.  Think image processing, shopping carts, database administration, Monte Carlo simulations, app “glue,” process control, AI, custom testing, IoT facilitation, site scraping, backups, pipelining, log ingestion and more.  Even better, WebJobs are super-simple.  You can be up and running in minutes, but more importantly, the “hard” stuff rarely takes more than hours.

If you work with the cloud: run, don’t walk.