Building a New Online Exchange

Building a New Online Exchange

The technical process and business context of setting up a new online exchange.

By Eric Parker
January 2004

Creating a new online exchange requires the development of custom software, because the commodity definitions and processes used in creating and clearing commodities are unique to each market. The technical development effort must support the process of industry transformation, and provide tangible artifacts reflecting the core value of the exchange. This document outlines the general development process for the creation of a commodities exchange, and how that process relates to the industry transformation catalyzed by the exchange.

Commoditization

A true online exchange must have the capability of functioning in a bid/ask continuous clearing environment, which requires formal data structure definitions capable of being automatically matched. While sufficient for auctions, free text descriptions cannot be used to accomplish reliable automated matching. It is possible to support auctions in a bid/ask exchange, but an auction process cannot be extended into a bid/ask environment without commodity definition.

Besides formal data structures, definition of a commodity also captures the process of creating a new commodity instance, and clearing a commodity instance post transaction. These processes are as much a part of the commodity definition as the data structure representation. Capturing the commodity life cycle requires representing the exchange participants, and the processes used by the exchange to manage participant capabilities and actions.

At conclusion of the commoditization phase of development, the exchange software must support:

Creation and management of new exchange participants, and management of the exchange itself by an administrator.
Creation of new commodity listings from a seller.
Clearing of transacted commodities to a buyer.

These capabilities are verified by using the software together with actual buyers, sellers and administrators. Feedback from these participants may require changes or refinements to the software. When all participants regard the automated process as obvious and intuitive, the class of goods has been successfully commoditized.

The technical development steps required for the commoditization phase include:

what: why:

Data structure definitions: Includes full persistency support (database code), and a user interface (typically a web app) capable of supporting add, modify, delete, query, display, and data navigation capabilities. Includes messaging support. Required. Captures the data structure aspects of commoditization.

Business logic processing: Includes user interface screen flow and supporting application logic (UI independent) necessary to support exchange participant management and commodity life cycle. Required. Captures the process aspects of commoditization.

Authorization framework: Includes user interface authentication (logon), action authorization and data filtering, security model. Required. Protects the exchange from rogue clients or other attacks and implements exchange data safeguards. Secure exchanges should consider dual firewall configurations and one-way encrypted passwords.

Deployment definition: Includes which processing will be running on what servers using which software. Required. Allows hosting environment setup.

Branding and interface design: Includes graphics and default look and feel for the application, and tuning of each interface screen in support of human factors. Important. Human factors can be critical factors in exchange participant buy-in and must be addressed before bringing an exchange live.

Online help: This consists of a framework capable of providing helpful information to users for specific forms and/or the overall application, coupled with the help text content. Important. Empowers users to learn the system while optimizing the interface for simplicity.

Documentation: Includes descriptions of all components of the delivered system. At a minimum this must describe the data structures, processing modules, and authorization processing. Important. Provides more detailed information than a functional specification, and supports system evolution. Avoids "I can't remember why we did it that way" issues.

Automation Acceptance

After a class of goods has been successfully commoditized, the exchange can be brought online for initial acceptance. At this stage, the exchange must support as many of the existing business process aspects as possible, while simultaneously introducing the features needed to support the online exchange. The work done in the commoditization phase ensures that only the transaction automation will require any significant adjustment from the exchange participants, maximizing the perception of the exchange as an additive ability or increased efficiency.

It is important that initial acceptance be done as a process verification, with manual oversight of payment processing and transaction clearing. Once the process is proven and accepted, automatic transfer of money and goods can be activated. At the conclusion of automation acceptance, the exchange software is functional and ready for production deployment.

The technical development steps required for the automation acceptance phase include:

what: why:

Auction processing: Includes all required auction features as required to automate existing business processes, transaction processing, participant and exchange user notifications. Required. Fundamental function of the exchange software.

Clearing: Includes all required processing related to transfer of funds and goods. May require manual intervention prior to production release. Required. Fundamental function of the exchange software.

Logging and reporting. Includes all output necessary for compliance with regulatory bodies. Must minimally include the capability to trace system processing on any transaction or group of transactions. Required. Trace capabilities are typically a legal mandate. The extent and sophistication of reporting varies based on the market and exchange requirements.

Runtime control panel. An interface allowing live runtime modification of exchange parameters, and start/stop/suspend/resume of major exchange processes. Required. Runtime control (at least at the level of gracefully suspending all exchange processing) is a necessary safety feature.

External alert processing. The ability of the system to send external notifications (via email and/or event management software) in response to conditions or events within the exchange. Important. External alert capabilities are an important part of managing an exchange. This becomes critical in subsequent development stages.

Data caching framework. The ability to safely cache data in local memory and avoid unnecessary database access. Important. Caching is key to UI performance and system scalability. This becomes critical in subsequent development stages.

Industry Migration

Once an online exchange has been accepted, it enters a period of growth as exchange participants join and begin trading. At this stage the online exchange supplements, and begins to replace, existing business practices.

Technical issues shift away from development and into production growth. The steps involved include:

what: why:

Multi-server production deployment: The exchange running on a minimal but scalable configuration, typically two web servers, an application server, and a database server. Required. Unless a single-server system will definitely be adequate in the medium term, it is best to deploy a scalable configuration immediately to avoid scaling issues in subsequent phases.

Secure transaction processing: Automated payment processing, and any automated clearing processing that occurs as part of the transaction, must be online at the stage. Required. In most cases, any residual manual processes may be overwhelmed, and impede transaction tracking.

Backup and recovery processing: Full recovery from backup, and disaster recovery procedures must be in place at this point. Required. Most exchanges have to be able to recover in the event of a catastrophic server failure.

Independent security audit: A full security audit by an independent third party, and any associated security hardening work. Required. Avoids negligence liabilities.

Cache tuning: Optimization of data caching tailored to server configuration and processes. Important. System data performance is best addressed incrementally to allow for appropriate deployment scaling over time.

Industry Transformation

Once an industry has accepted the exchange as a part of their day-to-day business processes, the exchange can begin to transform the industry through conversion to a continuous clearing bid/ask marketplace. During this transition, it is important to support both auctions and bid/ask modes seamlessly so that the transition is driven by market forces and with minimum transition for the user.

The transition from auction to bid/ask provides immense value, but is not technically difficult. Steps include:

what: why:

Bid/Ask processing: Continuous matching and clearing transaction processing, and associated administrative support. Required. This is the primary goal of commoditization.

Statistics: Aggregate information from runtime accumulators gives insight into realtime trends within the exchange. Important. Runtime information is valuable for predicting realtime exchange trends, both for market and system administration purposes.

Reports: Runtime status reports, timed averages from runtime accumulators, market summary information, trend analysis reports. Important. The value of report information usually far outweighs the effort associated with creating it.

Industry Evolution

With statistics and trend information, the commodities exchange can become a source for derivatives, which in turn can be used to manage risk. This is a later-stage transformation whose value depends on the size of the market.

In summary, creating an exchange requires three major transitions:

Commoditization
Automation
Migration and transformation

When the business, marketing, and technical aspects of this process are coordinated, these goals can be achieved within appropriate timeframes and at reasonable cost.

Eric is a software architect who has worked on a several online exchanges. Prior to founding SAND Services, Eric was CTO at SportsFutures, and Principal Technologist at TruExchange. He can be reached at eric@sandservices.com

what:	why:
Data structure definitions: Includes full persistency support (database code), and a user interface (typically a web app) capable of supporting add, modify, delete, query, display, and data navigation capabilities. Includes messaging support.	Required. Captures the data structure aspects of commoditization.
Business logic processing: Includes user interface screen flow and supporting application logic (UI independent) necessary to support exchange participant management and commodity life cycle.	Required. Captures the process aspects of commoditization.
Authorization framework: Includes user interface authentication (logon), action authorization and data filtering, security model.	Required. Protects the exchange from rogue clients or other attacks and implements exchange data safeguards. Secure exchanges should consider dual firewall configurations and one-way encrypted passwords.
Deployment definition: Includes which processing will be running on what servers using which software.	Required. Allows hosting environment setup.
Branding and interface design: Includes graphics and default look and feel for the application, and tuning of each interface screen in support of human factors.	Important. Human factors can be critical factors in exchange participant buy-in and must be addressed before bringing an exchange live.
Online help: This consists of a framework capable of providing helpful information to users for specific forms and/or the overall application, coupled with the help text content.	Important. Empowers users to learn the system while optimizing the interface for simplicity.
Documentation: Includes descriptions of all components of the delivered system. At a minimum this must describe the data structures, processing modules, and authorization processing.	Important. Provides more detailed information than a functional specification, and supports system evolution. Avoids "I can't remember why we did it that way" issues.

what:	why:
Auction processing: Includes all required auction features as required to automate existing business processes, transaction processing, participant and exchange user notifications.	Required. Fundamental function of the exchange software.
Clearing: Includes all required processing related to transfer of funds and goods. May require manual intervention prior to production release.	Required. Fundamental function of the exchange software.
Logging and reporting. Includes all output necessary for compliance with regulatory bodies. Must minimally include the capability to trace system processing on any transaction or group of transactions.	Required. Trace capabilities are typically a legal mandate. The extent and sophistication of reporting varies based on the market and exchange requirements.
Runtime control panel. An interface allowing live runtime modification of exchange parameters, and start/stop/suspend/resume of major exchange processes.	Required. Runtime control (at least at the level of gracefully suspending all exchange processing) is a necessary safety feature.
External alert processing. The ability of the system to send external notifications (via email and/or event management software) in response to conditions or events within the exchange.	Important. External alert capabilities are an important part of managing an exchange. This becomes critical in subsequent development stages.
Data caching framework. The ability to safely cache data in local memory and avoid unnecessary database access.	Important. Caching is key to UI performance and system scalability. This becomes critical in subsequent development stages.

what:	why:
Multi-server production deployment: The exchange running on a minimal but scalable configuration, typically two web servers, an application server, and a database server.	Required. Unless a single-server system will definitely be adequate in the medium term, it is best to deploy a scalable configuration immediately to avoid scaling issues in subsequent phases.
Secure transaction processing: Automated payment processing, and any automated clearing processing that occurs as part of the transaction, must be online at the stage.	Required. In most cases, any residual manual processes may be overwhelmed, and impede transaction tracking.
Backup and recovery processing: Full recovery from backup, and disaster recovery procedures must be in place at this point.	Required. Most exchanges have to be able to recover in the event of a catastrophic server failure.
Independent security audit: A full security audit by an independent third party, and any associated security hardening work.	Required. Avoids negligence liabilities.
Cache tuning: Optimization of data caching tailored to server configuration and processes.	Important. System data performance is best addressed incrementally to allow for appropriate deployment scaling over time.

what:	why:
Bid/Ask processing: Continuous matching and clearing transaction processing, and associated administrative support.	Required. This is the primary goal of commoditization.
Statistics: Aggregate information from runtime accumulators gives insight into realtime trends within the exchange.	Important. Runtime information is valuable for predicting realtime exchange trends, both for market and system administration purposes.
Reports: Runtime status reports, timed averages from runtime accumulators, market summary information, trend analysis reports.	Important. The value of report information usually far outweighs the effort associated with creating it.