I needed this utility to save 1 minute bar data tick data in a form where our AI / ML prediction engine could access it. I needed the history but even more important I needed the something that could deliver new ticks to the prediction engine fast as they became available.
I cheated with this utility and open the file in shared mode. This allows my Lua code read new lines as they become available as if the data was delivered by a more traditional pipe. I tested the lua code which reads to the end of the file and then recognizes when new data is available simply by continuing the read process. When no data is available it returns immediately so you can do a very low cost poll without having to re-open or re-read any part of the file. When I move to linux I can replace the file with a named pipe and eliminate the overhead of writing to disk.
Not a elegant architectural approach but it does keep the interface between my proprietary code and the 3rd party cAlgo product as thin as possible so I can change brokers easily in the future.
I was able to get it to deliver 1 minute bar data back to Jan-2011 which is the best source I have found for that much data. Still has a flaw in that I could not figure out how to calculate actual transaction volume for the bar but will comeback to that when it is critical. .