The following is an excerpt from my paper on Oracle messaging technology.
E-BUSINESS PROCESS INNOVATION
WITH ORACLE ADVANCED QUEUING TECHNOLOGY
Anthony D. Noriega, Independent Oracle Consultant, MSCS, MBA, BSSE, OCP-DBA
EXECUTIVE OVERVIEW
The advancement of contemporary information technologies has gone well far beyond imaginable boundaries from a decade ago. Internetworking has set the web as an invaluable resource, by changing business models at a fast pace, as new technologies enable their pragmatic development. In particular, this paper focuses on a discussion about advanced message queuing technology, by introducing the most relevant technical and business concepts, their usage, and the criteria that apply to their practical implementation. With the integration of business information systems, advanced message queuing models allow for a robust consolidation of Service Oriented Architecture (SOA) and Web Services applications, emphasizing reliability, and information privacy and security, and consistently enriching the steady growing information integration business landscape.
INTRODUCTION
The expansion of Middleware-Oriented Architectures (MOM) has lead to business models emphasizing robustness, persistency, reliability, and enhanced data protection via both information privacy and security.
This document introduces both technical and business concepts and models highlighting the following objectives, namely:
• Stressing the relevance of inter-process communication imposed by Oracle’s Advanced Queuing, by pointing out the related protocols and infrastructures.
• Positioning Oracle as the de facto leader in the database/message queuing integrated market, and a direct consequence of its comprehensive applications model sustained competitive advantage.
• Providing a brief discussion on programming techniques, methodologies, business models, with a concentration in requirements.
Therefore, it is possible to claim that the age of information integration has arrived, as such in e-Business, Oracle brings its dynamic queuing models to the concept of Enterprise Service Bus, which congruently positions the database leader in a new dimension in the integrated database/message queuing market, envisioning a new area of business process innovation.
THE TECHNICAL FRAMEWORK
CONCEPTS
The following concepts are fundamental to understanding Oracle’s message queuing technology:
QUEUE
A First In First Out (FIFO) data structure, i.e., specifically, a type of list. In advanced message queuing technology, queues can be persistent or non-persistent; normal or exception; transactional or non-transactional; with or without priority; regular messaging or buffered messaging; and based on their payload datatype, they can be of ANYDATA , XML, LOB or RAW (default) data types.
MESSAGE QUEUING
A technology that allows sending messages via a virtual channel from one queue to another queue or a database link, allowing information integration in different e business models.
HEADER
A set of features associated with the message set atop and accompanying its payload, by passing the first and most necessary information identifying the message.
PAYLOAD
The actual content of the message placed in the queue, i.e., the message content itself.
CHANNEL
A virtual communication path tuning a queue connection, sometimes called a topic. The channel is more relevant when using foreign queues, such as Websphere MQ or Tibco.
PORT
The port associated with the queue listener and the involved network connection.
ENQUEUE
The activity through which a producer application places a message onto a queue. The enqueue process is the mechanism producing the message and setting it into the queue.
PROPAGATION
The process through which a message stages and is moved from a producer queue, where it has been enqueued, to a destination queue, where is consumed by a consumer or recipient. Propagation is sometimes called Staging generically and imprecisely, which is not significant, for it does not convey the idea of travelling from one queue to another one.
DEQUEUE
The process through which a consumer application removes a message from a queue. Usually after dequeuing from a normal queue, the mechanism places the removed (consumed) message onto an exception queue. Therefore, the dequeuing process actions are associated with the mechanism consuming the message.
PRODUCER
An application placing messages onto a message queue, i.e., performing an enqueue operation.
CONSUMER
An application that removes or consumes a message from a queue, i.e., it dequeues the message from that queue.
RECIPIENT
An explicitly designated consumer. The term is widely used in publish/subscribe mode when utilizing multicasting, and more explicitly when an inbox or outbox is explicitly established for the recipient.
AGENT
An intelligent connectivity entity associated with a listener, a database link, and a relevant queue acting as a liaison to database and both local and foreign queue technologies. The agent establishes a binding relationship with a queue manager as well, via a protocol such as IPC using the gateway-established channel.
QUEUE MANAGER
The queue administrator handling inbound or outbound traffic through a established gateway interconnect channel. The concept is usually associated with foreign queues as well.
STREAMS AQ
An Oracle technology integrating message queuing and capture-propagate-apply downstream replication, event/alert management, and workflow models.
ARCHITECTURE
QUEUE MODELS
Based on the relationship cardinality model used, it can be classified as follows:
QUEUE MODE DESCRIPTION CAPABILITY
Peer-to-Peer (P2P) The message is sent from one queue to another or to foreign queue via a database link. There could be more than one receiving queue but only one can dequeue the message at one time. Only one consumer application can dequeue the message.
Publish-Subscribe This is a one-to-many model, which uses a subscription for recipients. When the number of consumers is unlimited it is called broadcasting. When that number of recipients is limited it is called multicasting.
Table 1. Cardinality-based Queue Mode Models
With the introduction of the Enterprise Service Bus concept, application architectural consideration can project the Model View Controller (MVC) more closely by breaking them down into domain, data model, and data source (Model); its presentation (front-end view); and its controller or mediator, thus establishing architectural fundamentals for grid computing implementation.
ADMINISTRATION AND MANAGEABILITY
According to the available Oracle literature, an application (producer) can enqueue messages that represent events into a queue explicitly, or a Streams capture process can capture database events and encapsulate them into messages called LCRs, resulting from DML or DDL changes. Propagations can spread messages in a stream through multiple queues. Finally, a user application (consumer) can dequeue messages explicitly, or a Streams Apply process can dequeue messages implicitly. Similarly, an Apply process can re-enqueue these messages explicitly into the same queue or a different queue, if necessary.
A database schema user can be a conventional user with the AQ_USER_ROLE privilege or a privileged administrator with the AQ_ADMINISTRATOR_ROLE privilege.
Similarly, Oracle10g handles and tunes the qmnc master process and its qxxx worker processes, in particular when the AQ_TM_PROCESSES initialization parameter is not set. Besides, the STREAMS_POOL_SIZE parameter default setting is 10% of the shared pool size.
An agent is established in connection with an extproc listener and a gateway home, when foreign queues or heterogeneous services are in place. The configuration of the LD_LIBRARY_PATH, and the installation of the Oracle Procedural Gateway are required with MQ and Tibco, while the Oracle Transparent Gateway is needed in Heterogeneous Services, like Microsoft MSMQ and SQL Server. The Oracle Procedural Gateway will normally allow for the start of the gateway without further configurations in a peer-to-peer mode, but it requires, otherwise, specific gateway home settings for each foreign queue established concurrently.
Among the most important PL/SQL supplied packages for this purpose, we can include DBMS_AQ, DBMS_AQADM, DBMS_AQJMS, DBMS_MGWADM, and DBMS_MGWMSG. Equivalently, a good number of Java Messaging Services (JMS) Java supplied packages is available and can be used concurrently or in lieu of PL/SQL.
SECURITY FRAMEWORK
Oracle’s queuing security framework is based on Streams rule-driven security at the object, schema, and global levels. However, it can also provide Virtual Private Database (VPD) and security can be further enhanced with realms set via Oracle Database Vault. Encryption models based on asymmetric authentication and the Public Key Infrastructure (PKI) can allow Oracle PL/SQL supplied packages such as DBMS_CRYPTO encrypt payloads prior to being enqueued and allow for their decryption via the public key upon consuming the message at the target queue (dequeue). This is widely supported primarily via ANYDATA, XML, and LOB data types.
ADVANCED FEATURES AND CAPABILITIES
Among the outstanding features and capabilities available in Oracle10g, it is possible to list the following, namely:
• Protocol Support including LDAP, IDAP, XA, SNMP, SOAP, TCP, and SMTP, among others.
• Payload transformation and encryption
• Reliable message buffering (messages residing in shared memory)
• Intelligent payload/content browsing for filtering or routing.
• Queue browsing without consumption
• Queue browsing without payload removal for auditing, logging, or non-repudiation.
THE BUSINESS FRAMEWORK
CONCEPTS
MESSAGE QUEUING
A modern technology, alternative to Remote Procedure Call (RPC) from the business process perspective, through which messages are placed in a queue (enqueue process handled by a producer application), and then propagated via a virtual channel to another queue, and finally consumed by a consumer application (dequeue process). A message can have a transaction or non-transactional nature. Although message queuing usually offers asynchronous services, implementing and customizing synchronous propagation models is also possible in distributed environments, in particular, utilizing the XA open systems transactional standard and the Two-Phase Commit (2PC) protocol approach.
DATA REPLICATION AND LOGICAL CHANGE RECORDS (LCR)
Replication is accomplished via a model through which transactions are captured, propagated, and applied from one instance onto another, being foundational for the Streams AQ downstream replication model. Streams-based replication defers from Advanced Replication (AR) in that it provides improved privacy, higher reliability, flexibility, and control. It usually relates administration and manageability from the producer application, while AR allows for a job-driven materialized view-based pushing and pulling of data, in various models such as peer-to-peer, multi-master, and chained replications –among others, the equivalent AQ downstream model may require the usage of several queues or intermediate queues. Today’s trend is to migrate or upgrade from Advanced Replication (AR) to Advanced Queuing (AQ), or more precisely, Streams AQ. This high-availability solution can also be used to implement a logical standby database.
DATA PROTECTION
Each queuing model provides a robust approach to protect information by providing information hiding, security, and privacy. Unlike Advanced Replication, AQ message queuing technology provides optimal information hiding and a higher level or reliability, and distributed transaction control with Oracle-based, open standards, and third party protocols.
SERIALIZABLE DISTRIBUTED PROCESSING
An AQ business model stresses data protection, reliability, and high-availability, as needed, in relation to the location of the resources and the distributed control imposed by the model. This model can achieve synchrony or depend on serially asynchronous events that can also be controlled via a priority queue.
INFORMATION INTEGRATION
Information Integration emphasizes the e-business needs for systems to share information in a timely manner. This concept is emphasized by the need of a strategy to implement a successful Enterprise Service Bus, which can consistently match business and system requirements in today’s grid computing models, in terms of automation, pooling, virtualization, and provisioning of resources.
BUSINESS MODELS
DATA WAREHOUSE LOADING
This business model utilizes advanced queuing technology to provide comparable ETL, OLAP, and overall data warehousing loading processes. It is heavily used in the financial sector, in particular, in banking and trading transactional systems.
EVENT MANAGEMENT AND NOTIFICATION
This model focuses on capturing management and notifications events and propagating them for alerting, aggregation, auditing, or journaling usage, as provided in Oracle10g grid computing by the Oracle Process Message and Notification Service (OPMN).
WORKFLOW
This business model concept stresses the criteria for event aggregation into useful information from the statistical viewpoint. The workflow model is practical for marketing, sales, and resources and capabilities business process balanced scorecards’ reviews.
AUTOMATION AND BUSINESS EVENT MANAGEMENT
This concept emphasizes the need to delegate the human control to systems that can also audit and notify the status of related events.
MESSAGE QUEUING IN SCM, ERP, CRM
Through information integration, business models incorporate information sharing and replication primarily in Business-to-Business (B2B), and Business-to-Consumer (B2C), but also in Consumer-to-Consumer (C2C) models. Therefore, the comprehensive e business information flow chain entices the usage of advance queuing.
Message queuing provides a mechanism to attain optimal data protection through data privacy, encryption, and security via information hiding, where visibility is virtually autonomous and implicitly transparent to the queue and its methods.
AREAS OF APPLICATIONS
The SOA and Web Services infrastructures are ideal for a queue-driven MOM-based application to achieve robustness. The most common areas of application include banking, trading, insurance, financial systems in general, and real-time applications where information needs to propagate at a fast pace with high reliability, for instance, in medical information systems, and research facilities in engineering and physics. Similarly, strategic group partnerships are born and develop in a nearly natural way, where IBM and Tibco constitute Oracle’s closest partners. Notwithstanding, a research on the Advanced Message Queuing Protocol (AMQP) has taken place in the last few years, with a recent July 2006 announcement by members of the partnership namely Iona, iMatrix, JP Morgan Chase, and Cisco, among others. AMQP is intended to become a standard for the financial sector. Yet, as the largest enterprise software vendor, Oracle is also the de facto leader in the growing message queuing market.
Protocol Standards Achieving advanced message queuing higher consistent standards is a major goal for future industry development and strategic partnerships
Vendor Interoperability Based on the standardization of protocols, it is expected that vendor interoperability become more transparent, rather user-friendly, and much easier to administer.
e-Business Operational Effectiveness Business can achieve a much higher level of reliability minimizing downtime and optimizing profitability based on the effectiveness sourced by information integration and related business processes at a lower cost.
Information Privacy and Protection Sarbannes-Oxley and HIPAA standards emphasize and promote further information privacy and protection, as provided by Oracle’s Streams AQ advanced message queuing technology.
High Availability (RAC), Encryption, and Virtual Private Database Support Oracle advanced technologies in high availability, encryption, and Virtual Private Database can guarantee, standardize, and consolidate the reliability needed in message queuing. In particular, RAC is foundational in the large enterprise architecture and infrastructure.
Table 2. Oracle10g Queuing: Envisioning the Future Business Landscape
CONCLUDING REMARKS
Oracle Advanced Queuing and Streams AQ are consolidated technologies that allow for fast replication, event management and alert, replication and workflow at low cost and with great effectiveness.
Oracle, as the largest enterprise business software company, positions itself categorically as the leader in the database and message integration combined market. Unlike traditional methods and technologies, advanced message queuing offers a variety of resilient capabilities, which conveys a robust and dynamic information integration management paradigm. It is possible to state that AQ is the mantra empowering the information integration at the opening of the new millennium.
REFERENCES
[ 1 ] Gawlick, Dieter. Message Queuing for Business Integration, eAI Journal, October 2002.
[ 2 ] Goyal, Brajesh et Mishra, Shailendra. E-Business Integration Using Advanced Queuing, Oracle Corporation, Proceeding from IOUG LIVE, 2005.
