Trade surveillance of physical trading - first questions, what are you trading and where in the world are you trading it ?

Last week, the DCM blog as on considerations that could influence choosing an "in house" or vendor solution to trade surveillance needs. That discussion really focused on transparent markets (such as futures) much more than physical markets. The closing note was that this week's discussion would focus on physical markets.
The most important questions on choosing an "in house versus vendor data system for physical commodity market trade surveillance are:
​1. What physical products do you trade; and
2. Where do you trade these products?
Fairly simple but crucially important questions that may need some further explanation. The starting point is to understand that all  of the first trade surveillance systems were emerging from the securities markets - equities and such. In those markets, every product has its own singular product code. So, from a data analysis point of view, it is very simple to have all similar trades in the same analysis buckets - just put every trade with the same code in the same bucket. Voila, you have the filtering done. Shifting from equities to futures is just a matter of expanding your look up table to include a bunch of new product codes. And that was the basis for the first energy trade surveillance software (that market started to emerge in the late 2000's with CFTC settlements that required companies to install trade surveillance).  And that is where the roll out hit a screeching halt.
I was SME on one of the very first roll outs. The data model was agile and flexible - the problem was it expected all trades to have a product code. The fun really hit doing global oil where the refined products desk traded US diesel and hitting oil in varying sulfur contents, Europe traded gasoil and, at that time, Singapore was trading products based on centistokes (viscosity). These were all different terminology for nearly the same product. The end result was the necessity to create concatenations of data fields to assemble a "product code". The negotiations necessary to get agreement across desks extended implementation times by orders of magnitude. There were a number of comments on last week's posting that alluded to exactly these questions from others who were also involved in some of these early projects - they have lived through these problems and seen the issues.
Now, fast forward to today. If you trade power and gas in Europe, you likely trade on a Multilateral Trading Facility ("MTF") which has specific requirements under EU regulation. An entity created for regulation of the European energy markets, the Agency for the Cooperation of Energy Regulators - "ACER", published a taxonomy for power and gas markets back in 2015 with the onset of REMIT (Regulation for Energy Market Integrity and Transparency). This led to development of a system for physical energy market product references very similar to futures markets product codes. Voila, the data problem is greatly reduced.
In the US and Asia, however, the natural gas, power, and oil markets have not had the same structure imposed. Therefore, most companies have an idiosyncratic data structure for naming and time bucketing of physical transaction data. Add to that the fact that outside of Europe most regulators do not have access to ongoing reporting of discrete transaction level data as well as any access to order level data for physical natural gas and oil markets (in the US, order level data is available from the Independent System Operators for certain products) and the efficacy of vendor solutions for the physical markets can be severely hampered. 
That efficacy issue does not even address the fact that, unlike exchange markets with products codes, a US physical or power market deployment is likely to be a "build from nearly scratch" data problem that is based on your ETRM system data structure, your naming convention for products in your system, and the complexity of transaction structures you trade.
The metals and agricultural markets can have their own complexities. Think about lending precious metals and the start and end date intricacies of those trades. First, how do you create a consistent product code? Second, what regulator has jurisdiction over these trades and how would they have the data to look at them?
In agriculture, think of the variations in moisture, quality, even age of the grain - you may "simplify" those trades into a common grade and quality for risk control but are they really the same trade for trade surveillance? And, just like in metals, what regulator is looking at those specific trades? Wouldn't a more general in house system that looks at the book as a whole and potential physical/financial cross-market influences be a better fit? Is there a vendor system that thinks that way or is a more simplistic in house build more appropriate?
I still get calls today from trade surveillance software vendors in discussion with customers in US physical markets asking why the customer is being very skeptical regarding their claims to handle physical commodity trade surveillance in the US. I go through a very similar discussion as set forth above. The data issue of "building from scratch" the product codes has a real implication in the estimation of installation costs. As noted earlier, the product code issue can add an additional 4 to 6 to 8 month data reconciliation issue in front of any commencement of the implementation - adding significant hard and soft costs to the customer.
For those reasons, DCM believes that trade surveillance development in physical commodity markets - metals, agricultural products, or energies - needs careful consideration of whether a singular vendor solution for physical and financial products is appropriate or whether bifurcation based on underlying data between vendor and in house might be appropriate.