• Level 1 is for new, incoming data only. No history is maintained in Level 1. Once the data is pulled into the Level 2 area, the data in Level 1 is truncated. This aligns to the purpose of Level 1: Get data from the source systems as quickly and easily as possible.
• The tables are data source organized. By keeping the structure of Level 1 organized similarly to the source systems, it simplifies adding new sources in addition to troubleshooting and management.
• To increase the maintainability, the naming convention within Level 1 should be close to the source. The exception: it is a good practice to prefix the tables within Level 1 to identify the data source. You will see this concept in our examples below.

Example Introduction

The example used here will follow through future posts for Level 2 and the Dimensional Model. This is the foundation for those future posts. For this example the premise is fairly simple:

Our task is to build a Data Warehouse for Thingamajig Inc. which will provide analysis capabilities on orders and shipments. Thingamajig Inc. builds widgets and sells them through the web, sales team, phone, and resellers. All orders except resellers are shipped directly from our outsourced logistics partner. Resellers ship their own inventory. All of the order entry systems are separate systems. The Data Warehouse is the aggregation point for all order data in the company. Shipment data will come from our logistics partner and resellers.

There are a number of challenges we will have to address in our design. However, the majority of challenges will be addressed in Level 2.

Orders Design

We are going to keep the example fairly simple for each order entry system. We’ll only include a customer and product information. Obviously most order entry systems are far more complex, but we don’t want to get caught up in those details.

Web Orders Design

The Web Orders system is the newest system at Thingamajig Inc. There are tables for order header, order line, product, and customer. The orders contained in the Web system are our end customer orders.

Phone Orders Design

The Phone Orders system is the oldest in the company. This one includes tables for order header, order line, item, customer, and address. The orders contained in the Phone system are our end customer orders either taken over the phone or entered from the field sales team.

Reseller Orders Design

Reseller Order data is a standardized text file that we receive from all of our partners. Luckily Thingamajig Inc. was smart by forcing all resellers to use the same file format. The orders we receive from resellers are their orders for product, not the end customer order.

Orders Design Notes

• All of our sources for orders are structured differently
• The terminology used in each is different. For example, web calls our product a product, while the phone system refers to it as an item.
• The Reseller order information is minimal. It does not contain the ship to address. That information is stored in a spreadsheet on the Reseller Manager’s computer.

Shipments Design

Shipments are less complex in that we only have two sources for data: Our resellers and our manufacturing partner.

Reseller Shipments Design

Just like our Reseller Orders, the data we receive is in a standardized flat file format. Thingamajig, Inc. uses shipment information to recognize revenue, so it is critical that this data is accurate. The data we receive from resellers is fairly basic as the resellers manage tracking deliveries to customers, etc.

Partner Shipments Design

Our Partner shipments are provided by an electronic transfer directly to a database. There is more granularity in the data because we follow the tracking numbers to make sure our shipments are delivered to customers.

So there you have it. Designed for speed and maintainability, Level 1 is the most simple part of the Data Warehouse architecture. In our next post which will discuss the Level 2 design, the ideas will come together… so stay tuned!

Tune in next time for: Level 2 Design

Now, onto our high level Data Warehouse Design overview. There are many different approaches to this topic, almost to the point of passionate debate of “Great Taste…Less Filling!” As with anything, your needs will depend on the specifics of the project. The design we’re talking about below has a number of advantages for the majority of projects.

Before we discuss the benefits of this design pattern, let’s understand the above diagram a bit.

• DB/XML/EDI/Flat File: Mix and match the various data sources that are required to meet your customer’s requirements.
• Level 1: Data source organized drop zone for data coming from the various data sources.
• Level 2: Subject area organized data store used as the launch point for downstream systems (which includes our Data Warehouse and more).
• DM/Cubes: DM = Dimensional Model. Dimensional models are a special database design which is tuned for reporting and analytical systems. Cubes are precalculated views of the dimensional model.
• User Interface: The access point to the Data Warehouse for our customers.

Layers of Isolation

A close friend shared the following quote with me a few years ago:

The only thing constant in life is change. -François de la Rochefoucauld

I cannot stress this point enough. The business climate today is under ever increasing pressure by a pace of life that is moving faster than ever before. The result is continuous churn in products, services, and processes for a competitive edge. For the lucky teams building systems with tight deadlines and budgets, our designs must be flexible and adaptable to whatever is coming next. For traditional data warehouse designs, change can be a real problem. Using Layers of Isolation can ease the pain.

Layers of Isolation is accomplished through the organization and purpose of the data at the various stages of the Data Warehouse. Going back to the Level 1 definition, we know the data is source organized. This allows us to deal with new data sources, logic changes, or new functionality from our source systems without disturbing the rest of the downstream Data Warehouse. Once Level 1 changes are implemented, Level 2 and Dimensional Model changes can be subsequently implemented. The development effort can be executed in parallel for speed or sequentially to minimize system downtime.

Organized, Simplified, Standardized (OSS) Data Store

Level 2 is our organized, simplified, standardized data store. Traditionally Data Warehouses serve as a tool for reporting, but our approach is to also provide a data store for the company. In larger companies it is not uncommon to have multiple systems that do the same or similar functions, with no cross communication. The business does not care about disconnected IT systems, just a unified view of data. There is also more than just multiple systems. Maybe your business sells through a Direct and Retail channel and the business logic in the order and shipment systems is radically different between the two. Our goal is to standardize so that regardless of business logic, data source, terminology, language, currency, time zone, etc. we have a unified view of the company’s data. Think about the benefits of having a data store for all of our business critical information in one place. There will be a number of projects beyond our data warehouse that will benefit tremendously. Clarity is important, so let’s define Organized, Simplified, Standardized:

• Organized: Bring together similar data into one place that is easily identifiable and joinable. Joinable is important if we want the ability to match Orders with Shipments for example.
• Simplified: The source systems may use complicated or arbitrary column names. Simplified uses an easily identifiable meaning from the name. If Attribute14 is actually Delivery Date, well then, call it Delivery Date. Another simplification that happens is denormalization. Most transactional systems normalize data. It could be that order information is spread across 10 tables in the source systems. Our data store simplifies this by appropriately consolidating data together into fewer tables (optimally 1 or 2 in our Orders example).
• Standardized: Different business processes and source systems may not use the same terminology. For example, one system might say Promised Delivery Date another may use Estimated Delivery Date. From a business process perspective, what both actually mean is the date we anticipate our customer receiving their order. We might just call this field Expected Delivery Date in Level 2 to standardize the meaning. There may also be cases when business logic will be applied to data. Level 2 is the place to do it once so data is standardized.

As a final benefit of OSS (Organized, Simplified, Standardized), we can do thorough quality assurance on a large chunk of the business logic. This helps us deliver consistent and tested data to downstream systems.

Development Speed

The layers of isolation naturally lead to a faster development cycle. Once the initial design of each layer is completed, developers can work in parallel to complete each section. In a traditional Data Warehouse design, there is typically only one level before the dimensional model, known as staging. In our design, you could say we have two staging environments. This is true and yes, there is additional development work required. We touched briefly on the Level 2 OSS benefits. If another project needs Orders, Level 2 is the place to go. No need to understand the 10 different orders systems in the company. This advantage significantly boosts development speed for downstream applications and more importantly, new Dimensional Models in our Data Warehouse.

Example Time

You have a company that sells widgets through retailers, along with phone, and online orders. Unfortunately, the IT systems used for each of these sales channels do not communicate with each other. A number of problems are created from this. There is no single view of customers, orders, shipments, revenue, marketing effectiveness and more. What is worse, the company is massively expanding internationally and setting up complete new IT systems in each country. The CEO wants a dashboard to understand the health of the business. In our Level 1 layer, we might have tables such as:

• canada_phone_order_header
• canada_phone_order_line
• canada_online_orders
• canada_online_customers
• canada_retail_order_header
• canada_retail_order_line
• canada_customer
• mexico_online_orders
• mexico_retail_order_header
• mexico_retail_order_line
• mexico_customers
• australia_phone_txn_orders
• australia_customer

The above represents the various IT systems and their deployments to different geographies. You’ll notice the inconsistencies across systems and even within countries. Once in Level 2, we’ll have:

• Orders
• Customers

Look good? Within Level 2 we have merged all of the order and customer information into OSS Orders and Customers tables. Next time a new country is brought online, we just add the appropriate Level 1 tables and logic to translate to our OSS Level 2 Tables. All of that without any changes required from Level 2 to our Dimensional Model.

While this is a highly oversimplified example, hopefully it wets your appetite for the deeper discussions on the design layers. At this point, you probably have more questions than answers. Over the next few posts we’ll cover the various layers in detail.

Tune in next time for: Level 1 Design Details