Expert Consultant Search, Browse, Discovery, Geo-Context Elastic Search


Expert Consultant in Search, Machine Learning and Distributed Architecture

We provide consulting to solve difficult Search, Discovery  and Machine Learning problems.   We are experts at applying Elastic search and Solar to Geo-contextual search in very large catalogs and consumer referral services.    We  are well proven consultants who consistently deliver innovative solutions to difficult problems.

Our ability to operate at both the business and techlogy level allows us to improve execution velocity  for initiatives that are important to business executives.  This can help IT teams deliver fast and get ahead of the delivery curve.

We have worked in senior levels so we understand business problems and can help executives meet their goals

joeeatbayesanalytic phone number Linked In Continue reading

Precision VS Recall, a Net Trading Profit Perspective


Forex Micro Strategies available for sophisticated users

I recently started making  Forex Micro-strategies available for sophisticated users under the general terms that they agree to pay 15% of their net profit.   Those that pay will also gain the benefit of continued enhancements and new micro strategies.  Contact us for access. Continue reading

KNN and Ensemble for stock price prediction


Applying KNN to stock price prediction

© Jan-25-2014 Joseph Ellsworth All Rights Reserved   phone-number-for-bayes-analytic Continue reading

Nov 11

DEM – Digital Elevation Model work with Scala

n51e001-london-ukDigital Elevation Models are the height maps of the Earth taken by Satellites.  This data was produced using the Shuttle Radar Topography Mission (SRTM)

The STRM3 data is available for most of the world while the SRTM1 data is available for most of the USA.   The SRTM1 data provides finer resolution than the SRTM3.

The basic quick start and topography docs are OK but working code is always easier to use.

I have included Sample Scala to download the entire world wide set of SRTM3 files along with the SRTM1 files for the USA.  There is also sample Scala code to read the .hgt files and convert them into grey scale renderings.  Continue reading

Oct 17

Scala fetch stock bar data from Yahoo finance service

Scala Example to retrieve yahoo stock bar data from Yahoo finance service.  Saves  resulting BAR data in local CSV file. Fetches each year from server and saves in file name where yy is replaced
with the year. The .day indicates this is a day level bar. The Yahoo service does not offer anything smaller than day. Continue reading

Oct 15

Scala Resource Links Getting started

Learning Methodology

When I learn new programming languages I first choose a non-trivial programming problem and then develop pieces of that system in the new language then go back and re-factor as I learn more.   Solving a real problem with the language is the fastest way I have found to gain demonstrated competence in a new language. Continue reading

Jun 02

Launched new site Air Solar Water for solar powered water maker

We launched the new Air Solar Water site this week.

This site offers our flagship technology for Solar Atmospheric Water Generators.

Solar powered Atmospheric Water Generator. Air Solar Water model A2WH-83S

Also announced that the A2WH-83S solar Water maker is available to buy now.

The Air Solar Water technology is uniquely designed to operated in remote off grid locations where ground water is not available.  It’s salt based desiccant design and use of solar heat as the primary energy allows it to operate in dry locations even deserts.

a2wh-83s Solar water maker

Water for dry cabins

The Air Solar Water model a2wh-83s is a perfect fit to add water for dry cabins in remote locations or research sites where no ground water is available.

Feb 26

Lua Source to calculate Kelly Criteria Investment Level

Lua Source demonstration of using Kelly Criteria to figure out appropriate amount to risk for any single trade based on your current capital, odds of winning and win versus loss payout. Continue reading

May 04

Automate Successful Manual Trading Strategies for Share of Profit? Good Idea? Bad Idea?

I posted the discussion Automate Successful Manual Trading Strategies on linked-in  where I have over 3,000 followers hoping for a more lively discussion.        Please add your comments at linked in where we will have the full stream.

The essence of the question:
is it good idea to apply our  skills and existing trading code and use them to automate manual strategies other people who are trading successfully and get paid via a share of the profit from the automated trading?

It seems that many traders with  3 to 10 million of capital traded by manual strategies may not be able to afford the risk of  automating the strategy.  They will also be hesitant to incur an unbounded maintenance overhead.    We can reduce labor costs by leveraging our code base of existing automated strategies and associated plumbing and connectivity.    In addition we are experts in exactly this kind of complex automation and really enjoy complex difficult problems and find trading algorithms absolutely fascinating.   I like business models with profit tied to the results of our activity rather than labor expended.

Continue reading

Apr 24

FX Robot trading results 4/5 to 4/24 Long only

These are the first 3 weeks worth of live cash trading results  occurring after our last change to the  EMA-Cross long Forex trading robot or Mini strategy.   Contact us for access

As many of you already know I think back-tests are great for optimization and hinting at potential but making predictions based on live market conditions and trading those predictions is the first time you really know if a strategy will perform.  Virtual is OK but trading live cash if better.

EMA-Cross-90-day Live Cash

Results 4/6 through 4/24 –  Starting Balance $244.51  with net profit of 13.71  represents gain of  5.6% for a period covering 19 calendar days or 63% of a trading month.     During this time it performed 20 trades with 15 wins for a win rate of 75%.     The largest win was net of $4.22 with largest loss of $2.24.   The average win was $1.54 while the average loss was $1.90.   This was made while the strategy was limited to risking 1% of capital per trade.   Risking 5% of capital per trade would have yielded closer 30% profit for the same 19 day period.


Continue reading

Apr 16

Talk to your tax specialist before making your first Forex trade

For those of you who are testing our Forex Mini  strategies you  need to talk to your Tax professional before you make your first trade during the fiscal year. There is an internal election you need to make and if you don’t you can be stuck with a tax treatment you do not like.    Continue reading

Apr 10

Mini-Strategy FX Results week April-5th 2015 – 68.8% win rate

This is a milestone week for our EMATrend Micro strategy which we are making available on a net profit sharing basis for a small number of sophisticated traders.   These strategies are 100% automated so once the user sets the configuration parameters they trade automatically without human intervention.

We finished the phase 1 short strategy with preliminary optimization the prior week so this is the inaugural week of trading both long / short for this strategy from a mixed account.  We use back-testing and WFA but the real proof is when you test a strategy against live market conditions. Continue reading

Feb 21

Feb 2015 Forex Trading Results for Quantized Forex Prediction Engine

Quantized Forex Prediction Engine up 3 for 3 Weeks when trading live data a total of 193 PIP gain.

25 trades with 17 wins for the test period Trading long only EURUSD Forex.

I started test trading our new Quantized Forex prediction engine using Live data during the first week of Jan-2015.   It took a few weeks to get things tuned and basically working.    I started a more formal test on Jan-29th where we keep things frozen for at least 1 week at a time on one account.     Since that time we have experienced 3 weeks of profits taking only long positions in a market that has been flat and down.    The ability to produce a profit during a down market indicates a promising future. Continue reading

Jan 03

Ideas for Automating Naked Trading using ML Techniques

Walters Naked Trading strategy is centered around a concept of investor fatigue and how to detect when the current trend is exhausted  and is likely to reverse. I have not tested the strategy  but I think it provides a nice framework for a well bounded ML problem that could be useful for the Apply Machine Learning for Investing meetup group.

I watched this presentation during my mandatory exercise time yesterday. It seems like a well bounded but non-trivial way get started testing various ML ideas. I added my ideas about how to think about automating for ML below. Continue reading

Dec 12

LuaJIT Access 20 Gig or More of Memory

Access 20 Gig or more from LuaJIT while coding in native Lua  and minimizing GC speed penalties.

I started using LuaJIT© after first using F#,  Python, Julia and C  for stock and Forex related predictive work.     I  am always on the lookout for a language that is high speed as close as I can get to C without having to write in low level C all the time.

Lua is a language that feels somewhat like a cross between BASIC and Ruby and has been around for a long time.   Lua may embedded or used stand-alone.  It has been embedded into many games,  entertainment consoles and other devices as a scripting language.    The LuaJIT  is a new  compiler technology and takes what was already fast  as an interpreted language and in some of our tests made it run over 20X faster with a few tests reaching 80X faster.

LuaJIT seemed like the ideal combination since it provided a language any ruby or python programmer would find readable  with fast start-up times,  excellent run-time speeds and good error messages.   Continue reading

Dec 10

Sourcecode to save Forex Bar data to CSV via cAlgo bot

I needed this utility  to save 1 minute bar data tick data in a form where our AI / ML prediction engine could access it.     I needed the history but even more important  I needed the  something that could deliver new ticks to the prediction engine fast as they became available.

I cheated with this utility  and open the file in shared mode.  This allows my Lua code read new lines as they become available as if the data was delivered by a more traditional pipe. I tested the lua code which reads to the end of the file and then recognizes when new  data is available simply by continuing the read process.   When no data is available it returns immediately so you can do a very low cost poll without having to re-open or re-read any part of the file.   When I move to linux I can replace the file with a named pipe and eliminate the overhead of writing to disk.

Not a elegant architectural approach but it does keep the interface between my proprietary code and the 3rd party cAlgo product as thin as possible so I can change brokers easily in the future.

I was able to get it to deliver 1 minute bar data back to Jan-2011 which is the best source I have found for that much data.   Still has a flaw in that I could not figure out how to calculate actual transaction volume for the bar but will comeback to that when it is critical. .

Contact us

Dec 06

Sourcecode to Download Forex Tick data to CSV via cAlgo

I needed this to save tick data in a form where our AI / ML prediction engine could access it.     I needed the history but even more important  I needed the  something that could deliver new ticks to the prediction engine fast as they became available.    I cheated with this utility  and open the file in shared mode.  This allows my Lua code read new lines as they become available as if the file were a more traditional pipe.   Not perfect but I wanted to keep the code in cAlgo as small as possible so I can change brokers easily in the future.  It only delivers about a year of back tick data but that is still over a gig of data.

A version of this code is used in our Production Forex Trading by our Forex Prediction Engine.

Contact us

Dec 01

Lua jit tests faster than Julia for Stock Prediction Engine

Lua jit tests faster than Julia for Stock Prediction Engine

I started testing Julia as a possible alternative because Julia advocates claimed the interpreter loop was nearly as fast a C and it was similar in concept to Python which I love but which was too slow for our application.   I recently ran across a blog entry mentioning a new Lua Jit. I found it intriguing because Lua did quite well during our last round of tests.

Performance comparison Julia versus Lua Jit

Relative Execution Time. Lua Jit as baseline – lower is better  
Operation Lua52 LuaJit Julia Julia Typed Array
Parse File into Data Frame 2.42 1.0 5.64 5.64
Compute SMA(14) 2.81 1.0 6.87 0.70
Compute SMA(600) 33.32 1.0 80.00 1.30
Compute SMA_slice(14) 2.42 1.0 11.87 1.83
Compute SMA_slice(600) 33.32 1.0 15.52 5.90
Did not implement slice in Lua so re-used the timing from nested loop version.
Response times are in seconds.

Only 1 tested Julia operation was faster than Lua JIT

The only function where Julia out performed Lua Jit was in the SMA(14) all other items tested were slower.   I think the reason it did better in this instance is that the SMA function must allocate a new array with 71K rows to store the results. In Julia you can do this as a typed Array of float.   In Lua this is done as an append to list so it is allocating memory in little pieces. In the SMA(600) the Lua jit was faster again because it is doing more work compute in a tight loop relative to the memory allocation overhead.

Continue reading

Nov 28

Testing Julia for Predictive Engines, An early opinion

The programming language chosen for predictive trading systems depends on where you are in life cycle. If you want to produce code to validate ideas then the fewest lines of code to produce a working system tends to favor languages like F#, Python and Julia. Even for early ideas you sometimes have to deal with considerable data volumes so you still need reasonable speed.     If you are coding for live operations where speed is essential then C/C++ are the best choice especially when unplanned pauses from Garbage collectors are unacceptable.    Julia is a new language designed for speed in critical areas.  It isn’t perfect but overall I am impressed.
Continue reading

Apr 28

Article in Inside Market Data – Bayes Analytic Preps Price Prediction Engine

Article in Inside Market Data – Bayes Analytic Preps Price Prediction Engine

We have an article in Inside Market Data. Bainbridge Island WA-based startup Bayes Analytic is preparing to launch a new stock analysis and trading recommendation engine that leverages advanced trading algorithms and machine learning to predict future price movements based on past trading patterns.

See The article at Inside Market Data
Contact us

Continue reading

Apr 28

$1,000 to 4.2 million in 6 years with no leverage an entertaining back test?

The KNN + Ensemble engine is designed to take stocks you would like to own based on fundamentals and maximize profit by adjusting the entry and exit times for those symbols. It attempts to predict which way prices will move the next day and sells the stock when the price is more likely to drop and buys when the price is more likely to rise. It may hold the stock for many days or only a portion of a day depending on if it thinks the price will continue to rise. No statistical engine will get every call right but our goal is to get enough right that it improves probability of profitable trades.

A contact at a major finance company recently asked me to analyzed over 5,000+ symbols for a full 6 years period. The engine beat buy and hold by a substantial margin most of the time. The ROI from SYN yields was abnormally high. I found it entertaining and thought it would be worth sharing. Continue reading

Dec 08

Distributed Queue – DQ provides durable sequenced delivery of millions of records per second to distributed servers in near real-time

DQ is an essential component in large scale web or eCommerce sites that need high scalability,  ultra high availability and super fresh data.

Continue reading

Oct 16

Allocate Trading funds to strategies based on market conditions

Trading systems should allocate trading funds to  strategies to maximize trading profits in current market conditions.    Some strategies perform well in specific market conditions such as topping while others will do better during strong directional markets.  Shifting fund allocation to strategies matched to current market conditions will yield the highest profit.    Operating multiple strategies increases risk of duplicate risk exposure that must be controlled by cross strategy risk management rules.

Ed Thorp is credited with adapting the Kelly criterion otherwise known as the Kelly betting system to determine how much to invest in a given stock based on the confidence of your selection criteria.   He published this in Beat the Market.   There is a lot of published art for the Kelly criteria but a simple version is also described in Thorps book “Beat the Dealer”. Continue reading

Sep 22

Economic impact on Spain from U.N prediction 700 million people migrating due to water scarcity?

Has anybody done a financial analysis for the economic impact implied by the U.N prediction of 700 million people migrating due to water scarcity as it relates to Spain?

Continue reading

Sep 21

Continued Fed easing is devaluing savings of middle america

Continued Fed easing helps the wealthy while damaging American middle class.

In the article U.S. Treasury Exceeds Debt Limit by $39 Billion with Accounting Trick –  Terence P. Jeffrey  explains a 0.01% rise in interest rates costs the Fed $3 billion in losses.

By artificially holding interest rates low the Fed avoids  multibillion dollar losses while the US Government can borrow at abnormally low rates.    Is fed easing really to help the economy or is it a way for the Government to massively devalue accumulated debt so they can avoid working on a rational budget? Continue reading

Sep 14

Trading losses caused by fake stock transactions

Watch for “Trade fake” stock transactions trying to trick you into stock trading losses that can be avoided Continue reading

Sep 13

Believable Test Trading must be done with real money

I am skeptical of back-testing until confirmed by testing with real money.

I believe the best test trading is done using a limited pool of real money conducting real trades that fully exercise the full chain of brokers and technology.   Other approaches seem to provide good data that can help during the development process but I have personally experienced too several instances where something works slightly different and allows the back-test to show results better than they can really deliver. Continue reading

Sep 12

Optimize Retail Marketing ROI using Bayes Analytic

The Bayes Analytic engine provides a phenomenal ability to select customers that have a high probability of responding favorably to a given offer.     This kind of tuned campaign delivery would allow you to use the same marketing investment but run many more campaigns each of which is delivering additional revenue.     From the customer perspective you are presenting better  offers with products and services they actually want to buy so they pay more attention to your future communications.

Bayes Analytic is interested in partnering with leading consulting firms on behalf of their major customers who need to improve the number of closes per customer contact.
Continue reading

Sep 11

Obey your Trading and Risk control rules

In an earlier blog entry I defined a stock strategy as a combination of   Algorithms + Goals + Symbols + Trading Rules.    It is tempting to take the output of the Bayes Analytic engine and apply the recommendation without obeying the rules.     Based on results from test trading earlier this year this is a very bad idea which can cause what is profitable strategy to be less profitable.      This concept is also draws from ideas described in  “When Good traders Loose“. Continue reading

Sep 09

Algorithm versus Strategy

I consider  Algorithm and Strategy as distinct concepts but in some language they could be interchangeable.


I consider an algorithm as a specific set of logical steps which are generally encapsulated as computer code that operate the same way and will give the same results when provided the same inputs every time they are executed unless the underlying logical steps are changed.

Continue reading

Sep 06

Distribution Rank as option Purchase filter Maximize option trading wins

Wins at option trading can be maximized by using Distribution Rank as option purchase filter.

When using Distribution Rank as a option purchase filter the best rate of winning for SPY Call options occurs when the 10 day distribution rank is 70% to 85%.  The winning rate is at this level can be 3 times higher than when the distribution rank is between 15% to 45%.   This shows the buying at the lowest possible price will not produce the best ratio of option trading wins. Continue reading

Sep 04

Why Language Performance Matters & Some Measurements

An optimization pass that takes 10 minutes in one language could take 15 hours in the slower language.

I work from the philosophy that I want the highest performance I can afford but this is traded off against development costs and delivery time because a great solution 6 months later  may be less valuable than a good enough solution 6 months sooner.

Continue reading

Aug 10

Bayes Analytic Engine is really Bayes+ Morphed for prediction at high volumes

I started out to building a simple Bayes classification engine what the industry would normally consider a naive Bayes classifier.    In the process of testing and enhancing for stock trading I needed the same world view but very different engineering.     In the process it evolved in different directions with many changes away from the naive classifier. Continue reading

Jul 05

How Day purchased and Week of Month affects option trading sucess

The Bayes Analytic engine is designed to predict probability of success for a given offer to meet a goal.   In some instances the goal could be will a particular customer buy from a specific offer.   I originally designed the stock prediction Engine to answer the question of which recent trades have the highest probability of meeting a specific goal such as increasing in value by 23% over a 7 day period.   The idea was that if we treat every option trade like an offer if we could isolate the set of offers which provided the highest probability of reaching the goal then we could buy those and skip the trades which offer a lower probability of delivering a profitable trade. Continue reading

Jun 15

How Bayes Analytics can benefit companies in the Travel Industry

The Bayes Analytic engine is nearly ideally suited for use in the travel industry companies like who have a long term relationship with their customers and who need to continually deliver more value and save more time for their customers.    While the engine is optimized to determine the probability a given user or deal will respond favorably or meet a pre-specified goal the same feature can be reversed so it can calculate which offers or goals have the highest probability of success. Continue reading

Jun 02

Bayes Analytic for Communications Companies

I have been working on analytic components which may provide benefits to AT&T wireless or similar companies.   AT&T is particularly interesting because they have multi-year relationships with their customers but do such a poor job of targeting communication (I say this based on personal experience).  AT&T marketing is actually alienating customers when they could have been adding value and building loyalty.  Continue reading

May 05

Stock recommendations engine with Machine Learning and AI based optimization

Introduction to Bayes analytic Stock recommendations engine

The stock recommendations engine delivers maximum stock trading profit by finding the subset of current buying opportunities which offer the highest probability of producing the desired profit.    It combines hundreds of statistical measurements with proprietary trading algorithms to deliver a high degree of accuracy and trading profits.     Advanced trading algorithms  allow the system to adjust trading recommendations throughout the day to adjust for changing market conditions. Continue reading

May 04

Bayes Analytic Retail Marketing with Machine Learning

The Bayes Analytic engine is a  machine learning classification engine designed to predict which stock offers a have the highest probability of success.  This can be applied to  price movement or customer behavior.     The engine can be of benefit to organizations that maintain long-term relationships with their customers.  It can ensure more customers receive offers for products they will find valuable which can increase the sales per contact and cause customers pay more attention to future communications. Continue reading

Mar 01

Why not use existing trading engines, R or other analytic tools

Why did you need to build the Bayes Analytic engine when so many analytic engines are available on the market?

The simple answer is that I has specific requirements and could not meet those requirements with the software I had available.     The business answer is to provide unique competitive advantage to the hedge funds we partner with. Continue reading