Trying to understand the Oracle Reference Architecture for Information Management

Last month I have been attending the RittmanMead BI Forum 2014. In the wrap-up I mentioned a presentation by Andrew Bond & Stewart Bryson. They had a very nice presentation about the Oracle Information Management Reference Architecture. This needed some further investigation from my part.

This blogpost is a first summary of the information I found online so far.

There is a complete reference library of IT Strategies from Oracle. Pay extra attention to the following docs:

  • Oracle Reference Architecture Information Management
  • Oracle Reference Architecture Business Analytics Foundation
  • Oracle Reference Architecture Business Analytics Infrastructure
  • Oracle Reference Architecture Service Orientation
  • Oracle Reference Architecture Security
  • Oracle Reference Architecture Engineered Systems

IT Strategies from Oracle

Next to that, Oracle has a few White Papers focussing on Information Management (Big Data & Analytics):

Oracle Information Management – Logical View

Let’s take a closer look, see the picture below. This picture has been copied from; the Oracle Information Management Reference Architecture. presentation I mentioned above. There are a few major components in the Reference Architecture

  • Data Sources
  • Information Provisioning
    • Data Ingestion
    • Logical Data Warehouse
  • Information Delivery

While putting together this blog post, Mark Rittman posted the following article(s); “Introducing the Updated Oracle / Rittman Mead Information Management Reference Architecture – Pt1. Information Architecture and the “Data Factory” & Pt2. – Delivering the Data Factory” on the RM Blog.

Oracle Information Management – Logical View

Data Sources

Within this Reference Architecture Oracle should be able to handle all sorts of data:

  • Traditional Enterprise Data (ERP, CRM, etc.)
  • Machine-generated /Sensor Data (Smart Meters, Equipment Logs, etc.)
  • Social Data (Twitter, Facebook, etc.)

Any Data - Any Source - Any Format - Different Data

The last couple of years / decades, Data has changed. We (as BI/DW-Consultants) have always looked at data in a Traditional (Structured) way. Now Oracle provides an Architecture to combine the old Data with the new (Big) Data. Whether there are 3, 4, 5, or whatever number of V’s, the most important thing is that you get value from your Data!

Data Ingestion (Loading)

Oracle provides / supports tools to perform Batch or (Near) Real-Time Data Ingestion.

Oracle GoldenGate & Oracle Data Integrator

Michael Rainey is writing an excellent series on how to load the Raw Data Reservoir (Staging Layer) and the Foundation Layer; Oracle GoldenGate and Oracle Data Integrator – A Perfect Match in 12c.

Cloudera Distribution including Apache Hadoop (CDH)

Hadoop is Open Source and Apache licensed. Cloudera provides an Open Source Distribution ( Cloudera CDH - Cloudera Enterprise Data Hub).

Oracle Event Processing (OEP)

OEP is a complete solution for building applications to filter, correlate and process events in real-time. Check the Data Sheet here.

Logical Data Warehouse

There are a few different Layers:

  • Raw Data Reservoir (Staging Layer)
  • Foundation Layer
  • Access and Perfomance Layer

Depending on the requirements and the tooling one can decide to skip or combine the different Layers. Oracle has the tooling to load the various Layers directly (GoldenGate).

The data is no longer only stored in the Oracle Database. The ‘new’ types of data require ‘new’ types of storage. There are different Data Stores for different purposes.

The Oracle Big Data Connectors can be used to integrate Apache Hadoop with Oracle Database Software.

Processing the data could be either In-Database (Oracle Database Options –  Advanced Analytics, OLAP) or In-Memory (Oracle TimesTen).

Information Delivery

Oracle is able to support proven answers to known questions via Oracle BI. Fast answers to new questions are provided by Oracle Endeca Information Discovery.

“The Oracle BI Foundation Suite provides comprehensive capabilities for business intelligence, including enterprise reporting, dashboards, ad-hoc analysis, multi-dimensional OLAP, scorecards, and predictive analytics on an integrated platform”

Oracle BI Foundation - Front-End

OBIEE makes it possible (Logical Layer) to skip the ‘Access and Perfomance Layer’ and source directly form the ‘Foundation Layer’. There are some very interesting presentations online around this subject.

“Oracle Endeca Information Discovery is a complete enterprise data discovery platform that combines information of any type, from any source, empowering business user independence in balance with IT governance. Now organizations can access the information they need, when they need it, to make business decisions they can trust.”

There is still a lot more to investigate, but for me this gives a little bit more guidance.

RittmanMead BI Forum 2014 – Wrap – up

RittmanMead BI Forum 2014

Last week I was (again) one of the lucky ones to join the Rittman Mead BI Forum 2014. This year we were served with a 3-day program. Just like the other years the program was varied. Varied in subjects  (Oracle BI EE, Oracle Endeca, Aggregation Persistence, Big Data, Oracle Exalytics, Oracle BI Cloud,  Oracle Information Management Reference Architecture, etc.) an varied in in style; Keynotes, one hour presentations and 10-minute TED-sessions. Next to that the possibility of a Masterclass around Hadoop Cloudera.

Just like all the other days the Rittman Mead BI Forum 2014 was an excellent place to network and exchange ideas. Next to the people I knew already I have met a lot of ‘new’ people. These days were a big inspiration for me. I do not have to be bored the coming year.

I was very impressed by the Masterclass of Lars George around the topic of Cloudera Hadoop. If you want to explore this subject more there are several options. Thanks to Alex Hansal for linking to these options.

It was interesting to hear how Oracle invests heavily in the Oracle (BI) Cloud. As a part of that a lot of new interesting Oracle BI functionality will become available in the Cloud first. Hopefully these functionalities will arrive on Premise shortly after.

Andrew Bond & Stewart Bryson had a very nice presentation about the Oracle Information Management Reference Architecture. This needs some further investigation from my part.

I often see clients struggle with reporting on Oracle eBS. There are a lot of alternatives. Truls Bergersen showed an alternative via the Oracle eBS extensions for Endeca.

Congratulations to Robin Moffat for receiving the ‘Best Speaker Award’ for the second time! Check his presentation; ‘No Silver Bullets – OBIEE Performance in the Real World’ here.

A big thank you to Mark Rittman, Jon Mead and their crew. Also a big thanks to Oracle for joining and for being so open about the Oracle BI developments and their visions and roadmaps. Thanks all for joining and sharing. That’s what makes the Rittman Mead BI Forum such a great event to be at. Hope to see everyone next year.

Tomorrow the US edition of the Rittman Mead BI Forum will start. If you would like to stay informed, you should check the Tweetchat.

After the US edition all the presentations of the RittmanMead BI Forum 2014 (Brighton and Atlanta) will be online. There is definitely some good stuff to have a look at!!

RM BI Forum 2014 - Tshirts

RM BI Forum 2014 Notes – Day II

RM BI Forum 2014 Notes – Day II

Find below some notes, links, etc regarding day I of the RittmanMead BI Forum 2014. These are just some notes and it’s by no means a re-cap of the complete presentations. Check the RittmanMead blog after the Atlanta edition of the RittmanMead BI Forum 2014. They will post all the presentations (both Brighton as well as Atlanta)

Drawing in a New Rock on the Map – How will Endeca Fit in to Your Oracle BI Topography by Truls Bergersen

Data Discovery
– Unknown Answers to Unknown Questions
– New Datasources

Hybrid search/analytical database
Key Value Pairs
In-Memory Analytics

Self-Service Provisioning – (Excel, JSON-Files, OBIEE Data Source)

Web Acquisition Toolkit

ODI KM – Integration Knowledge Module

Sentiment Analysis (only via Integrator tier) – Lexical (Dictionary)

Oracle eBS extensions for Endeca (light-weight Oracle BI Apps)

Real-time Data Warehouse Upgrade – Success Stories by Nicholas Hurt & Michael Rainey

Events —> ETL –> Cleanse –> De-dupe –> Summarize –> Dashboard
Streams & Oracle CDC —> Oracle GoldenGate (Journalizing – ODI)
OWB —> ODI 11g – 3R’s- Re-Asses, Replicate, Refine existing mappings

JMK Oracle to Oracle Consistent (OGG) Knowledge Module

Oracle Information Management Reference Architecture

Staging Layer -
Performance Layer (Dimensional Model – Star Schema)
Hybrid Layer

Real-time BI: An Introduction

GoldenGate and ODI – A Perfect Match for Real-Time Data Warehousing

Oracle BI Cloud by Matt Bedin

Sort of NDA, but it will be there……the Oracle BI Cloud

Oracle invests heavily in the Cloud. As a part of that a lot of new interesting Oracle BI functionality will become available in the Cloud first. Hopefully these functionalities will arrive on Premise shortly after.

Essbase within/without OBIEE – not just an aggregation engine by Gianni Ceresa

Essbase and OBIEE Aggregate Pesistance wizard

Essbase Studio

ASO (Aggregate Storage Option)
BSO (Block Storage Option)

CalcScripts

UDF – User Defined Functions
Essbase Writeback

Speed up RPD Development by Marco Klaassens

Speedup Delivering = Knowledge * Focus * Offering

Neo’s Voyage in OBIEE by Christian Berg

Blue Pill – It’s Known
Red Pill – It’s Undocumented (Use at own risk)

NQS Procedures as Physical Tables (ODBC) in the Physical Layer

Contact Christian directly, he might want to share.

Tuning TimesTen with Aggregate Persistence by Alistair Burgess 

Tuning TimesTen
  • RangeIndexType=0
  • TimesTen Data Types (TT_INTEGER, TT_SMALLINT)
  • RAM Policy
  • Compression
  • Index Advisor
  • Execute on Connect
  • Parallel Query
Licence costs (compared to the Oracle Database) could be interesting

RM BI Forum 2014 Notes – Day I

RM BI Forum 2014 Notes – Day I

Find below some notes, links, etc regarding day I of the RittmanMead BI Forum 2014. These are just some notes and it’s by no means a re-cap of the complete presentations. Check the RittmanMead blog after the Atlanta edition of the RittmanMead BI Forum 2014. They will post all the presentations (both Brighton as well as Atlanta).

Extreme Intelligence by Emiel van Bockel

Emiel started the day where he took us on a journey with regards to the Exalytics implementation @ CB Logistics. Exalytics is one of the Engineered Systems Oracle has.

An Engineered System is not a Plug ’n Play Environment. It needs to be Engineered. Only start engineering when you are sure system works.

Know your system (understand your data)

Do not use the wizards, but use your head/experience. Do it yourself.

If done right you own your own Extreme Intelligence System. The best in the world!

Times Ten Best Practice and Optimization by Chris Jenkins

Chris who is Senior Director, In-Memory Technology, TimesTen Development at Oracle gave us some very detailed insight into Oracle Times Ten.

OS Configuration

Shared Memory Segment

Huge Page considerations

Semaphores

Save storage (inline – out of line) by comparing Data Types between an Oracle Database and the Times Ten Database

Compression

Loading Times Ten via…..

Incremental refresh

Parallelism

Performance Optimisation

 OBIEE Performance in the Real World by Robin Moffat

Evidence based Design and Diagnostics

Create Time Profile – “Nose to Tail” —> Which component (OBIEE, Weblogic, Network, Database, etc.) is causing trouble.

Dive into the NQQuery Log – Query Logging is not evil! Check for more details here.

DMS metrics (obi-metrics-agent)

Grafana – Public live demo online

Must Read – “Thinking Clearly about Performance

Oracle BI Cloud by Adam Bloom

Sort of NDA, but it will be there……the Oracle BI Cloud

Oracle invests heavily in the Cloud. As a part of that a lot of new interesting Oracle BI functionality will become available in the Cloud first. Hopefully these functionalities will arrive on Premise shortly after.

Extreme Data Warehousing by Paul Oprea

A lot of discussion about Agile and Waterfall.One ting is important; Keep the user engaged and involved!

A Picture Can Replace A Thousand Words by Michael Rainey

Using pictures to vizualize the process will help in getting the requirements clear.

BPMN

About the Oracle DW Global Leaders Program by Reiner Zimmerman

http://www.oracle.com/technetwork/database/bi-datawarehousing/oracle-dw-global-leaders-brochure-345526.pdf

 Enterprise Big Data Architecture by Andrew Bond & Stewart Bryson

Oracle Information Management Reference Architecture

Discovery Lab

Foundation Layer – (Raw) Data Reservoir – Golden Gate  -> Mapping Physical to Logical (OBIEE) – No ETL, yet

Data Factory (ETL)

Enterprise Information Store

Event Engine

Data Ingestion

Information Interpretation

Agile Manifesto

Model Driven Development – Sandbox

Generate Logical Model Documentation (OBIEE)

Access and Performance Layer

Refactoring – Solve challenges in the Front-End. Working report -> Then move down to RPD or ETL

TDWI Chicago 2013 Keynote: Big Data, Bigger Impact

RM BI Forum 2014 Notes – Cloudera Hadoop Masterclass

Cloudera

The Rittman Mead BI Forum started off with a one-day Hadoop Masterclass, provided by Lars George.  As he messaged us the day before we have learned what Hadoop is all about, what its major components are, how to acquire, processes and provide data as part of a production data processing pipeline. To that effect, Lars advised that it would be useful to follow along the examples in the course and have an environment handy. That would allow us to experiment at our convenience during and after the class. He directed us to the following link; the Cloudera Quickstart VM.

Lars recommends the following: “Select the CDH5 version of the VM. Please select a virtual machines image matching your VM platform of choice. If you do not have a VM host application installed yet, you can choose from a few available ones. VirtualBox is provided by Oracle and a great choice to use. It can be downloaded here. Set up the VM application, then download and start the Cloudera Quickstart VM to run on top of it. It is as easy as that.”
Find below a few notes I took during the Masterclass.
Lars devided the Masterclass into four parts.

I – Introduction into Hadoop

  • What is Big Data? –  It’s not necessarily volume but also format and speed. Three V’s – Volume, Variety and Velocity
  • Introduction to Hadoop
  • HDFS
  • MapReduce
  • YARN
  • Cluster Planning

Hadoop is Open Source and Apache licensed — http://hadoop.apache.org
Many developers Cloudera, Apple
Contributers
Many related projects, applications, tools
Hadoop is not a system but a set of tools, projects which work together. You should decide, for each part of the architecture, which tool you should use and how you would use it.
HadoopEcosystem

Hadoop where to get it?
Load, Process and Analyze data
Hadoop Concept – distribute data in the system.
Process the data where it resides
No network processing
High level code (java)
No communication between nodes
Data stored on different machines in advance

Map Reduce Data Flow
  • Map
  • Sort en Shuffle
  • Reduce

II – Ingress and Egress

Ingress – moving data into Hadoop (HDFS)
Flume  (Near Real-Time Pipeline)
  • Source
  • (File) Channel
  • Sink —> poll, collect and write to eg. HDFS
Apache_Flume
Transfer data between Relational Database (Oracle, Terradata, Sql Server, etc.) and HDFS
Oracle Database Driver for Sqoop – OraOop by Quest 
FIle Formats important to keep in mind when you want to get the data out again.
Simple File versus Container (Structured) File
Parquet vs Google Dremel —>
BI Integration
  • Sqoop
  • HDFS Connector
  • ODBC/JDBC

III – NoSQL and Hadoop

ACID (atomicity, consistency, isolation, durability) 

IV Analyzing Big Data

  • Pig
  • Hive  (HiveServer2 instead of HiveServer1)
  • Impala
  • Search – Lucien
  • Data Pipelines (micro –  macro)
  • Oozie (Workflow Server)
  • Information Architecture – Where / How to store data and how to secure this structure
  • Spark (Java, Python, Scala compile into code)
I think Lars could have talked about Hadoop two more days (with or without sheets). Hadoop is all about making choices. There are similar tools, projects, concepts, etc. All depends on what you want to achieve.
Although this Masterclass was very informative, I still struggle to see the use case at this moment. A lot of my customers are still struggling with their ’normal’ data……

RittmanMead BI Forum 2014

 

RittmanMead BI Forum 2014

 

Today the 6th (my 5th) RittmanMead BI Forum takes off in Brighton. As usual the guys at RittmanMead have been able to put together a variety of sessions (Presentations, TED session, Keynotes, Debate) around OBIEE through to Oracle Endeca, Oracle Data Integrator and Oracle Essbase and In-Memory Analytics Databases.

For me the RittmanMead BI Forum is an opportunity to meet with the Premier League in Oracle BI&DW.

Check the RittmanMead website for the agenda and more details on the Lars George Cloudera Hadoop Masterclass – RM BI Forum 2014, Brighton & Atlanta

I will try to keep up a diary of the following three days.

Check back here to stay posted. If you want some liver coverage please check the following hashtag on Twitter; #biforum2014.

Mobile Analytics – Native, Web, Hybrid

Last week I had a discussion with my colleagues over @Scamander during one of our Tech Sessions. The subject was (Oracle) Mobile Analytics, which is one of the Big Data and Analytics Top Ten Trends for 2014. During the discussion we were discussing the difference between Native Apps and Web Apps and the necessity of a Native App when you can built / use  a Web App. Because of a time constraint we were not able to finish our discussion.

My colleague; Daoud Urdu pointed me to the following presentation; ‘HTML5 and the Journey to the Modern and Mobile Web‘. This gives some valuable background to Mobile Application Development.