About a year ago, I wrote about how to think about a monitoring solution in the SOA/BPM world . The focus was on the holistic thought process for an enterprise wide monitoring strategy. My experience at the time was (and still is) that IT often doesn't understand the difference between reporting and monitoring. A monitoring solution often gets picked just because it sounds like a cool new technology (not to mention it makes the resume attractive) and business likes to idea of having pretty looking dials, charts and graphs.

Monitoring and reporting are two different concepts. A business solution may utilize both to satisfy the business requirements. Keep in mind the following differences between them when picking one versus the other:

• Reporting is a data driven solution, which shows information about the business. This makes the report a snapshot of some business information at a given point in time. On the other hand, a monitoring solution is an event based solution, which shows what is happening during business execution. This makes monitoring solution a 'view' into the business. The view is not a time based snapshot. It tells you what is happening now
• A report largely shows historical data. It's the information after the fact whereas a monitoring solution can show current information, historical information and even trends
• A report tends to be static. There are solutions where business users can generate ad-hoc reports. However, these ad-hoc reports merely given enhanced filtering or query capabilities on the business data. Monitoring, on the other hand, is much more dynamic. It allows insight into the business from different perspectives. Depending on the solution used, it can even allow the business to reshuffle things and dynamically change business process to respond to the monitored events. For example, a monitoring solution can keep an eye on the number of claims arriving in a particular processing center. If the number goes beyond certain threshold the business can route new claims to a different processing center on the fly
• The only source of information for reports is database. In case of monitoring, the information being monitored does come from the database from technical perspective but the source of that information could be an event generated by anything: a business object, a system, an application or even an entity external to the organization

Now that we know the difference between reporting and monitoring, next time we will look at an example requirement and decide whether reporting or monitoring makes the most sense.

WebSphere Message Broker (or message broker for short) enjoys a second class citizen status to many developers and architects - specially those that come from WebSphere Application Server and/or J2EE background. It took me a long time, many discussions and bunch of stressful and often heated debates to convince one of my clients that the message broker is and can be used as an ESB and it will work just fine with WebSphere Process Server. If I had failed, they would have used WebSphere ESB for all interactions with process server and message broker for everything else, essentially resulting in 2 ESBs and potentially defeating the purpose of the ESB. And this is not the only time I have seen this.

What amazes me is not the fact that people don't like message broker. It's the reasons they give for not wanting to use it. In the rest of this blog entry, I am going to list those reasons, which are really myths, and dispel them.

Message broker has poor support for web service standards. WebSphere ESB has better support. If you are focused on standards based interactions using XML, SOAP and WS* then go with WebSphere ESB.

This is not true. Message broker supports web services standards as well as if not better than WebSphere ESB:

• It has support for web services standards including Basic Profile 1.1, see http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/topic/com.ibm.etools.mft.doc/ac55890_.ht
• In WS-* standards, it has support for WS-Addressing, see, http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/index.jsp?topic=/com.ibm.etools.mft.doc/ac64300_.htm and WS-Security, see, http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/index.jsp?topic=/com.ibm.etools.mft.doc/ac55630_.htm
• On vanilla web services, message broker actually supports more SOAP processing nodes than WebSphere ESB. Broker soap nodes are listed at: http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/index.jsp?topic=/com.ibm.etools.mft.doc/ab00026_.ht
• Message broker even supports SOAP/MTOM and integration with a registry such as WSRR
• It has out of the box functionality to generate schema based messages from WSDL, which WebSphere ESB doesn’t
• It even allows you to make any text based message a SOAP message, get SOAP envelop in or out, get SOAP body out and so on – WebSphere ESB supports this but it doesn't have this capability provided out of the box in the development tool (integration developer
• It provides a lot of XML functionalities including XSLT that WebSphere ESB provides. See this link to get a high level idea http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/index.jsp?topic=/com.ibm.etools.mft.doc/ac67160_.htm

Message broker requires you to use proprietary things like ESQL. So code developed in message broker code becomes proprietary. WebSphere ESB is based on J2EE and SCA (Service Component Architecture) those are both standards. Mediations developed for WebSphere ESB could be ported to another standard platform in the future

Again, this is not true.

• You can use ESQL or Java in message broker. While ESQL is proprietary, Java is an open standard
• While SCA is a standard and some vendors are implementing it, portability across containers is still a dream. Plus, not every vendor wraps mediations in an SCA component. Also, packaging of SCA components (in jar and ear files) appears to be specific to WebSphere ESB and process server. In some cases, you have to extend a com.ibm class or use a com.ibm class behind the scene (e.g. binding classes). The point isn't that SCA is bad. The point is, although SCA is a standard, use of SCA in WebSphere ESB doesn't make ESB code portable

If you have J2EE developer adoption of WebSphere ESB is easier than message broker.

This argument does hold some water. WebSphere ESB is built on top of WebSphere Application Server. It uses ear and jar as deployment artifacts, it uses EJB and Web projects behind the scene. However, you should keep in mind that there is a learning curve involved in using WebSphere ESB as well. I would agree that the learning curve for a J2EE developer on message broker would be steeper.

The point of this blog entry is not to say that message broker is better than WebSphere ESB. Nor that the message broker should be chosen over ESB. The intent is to dispel some myths so it is not discarded for wrong reasons.

The world was a happy place when ESB was a pattern. We implemented the bus pattern using a combination of tools and technologies. Now, ESB is a product. And, if you are using IBM products, you have two of them: WebSphere Message Broker and WebSphere ESB.

Here is how webster.com defines 'bus': (1) a set of parallel conductors in a computer system that forms a main transmission path (2) a spacecraft or missile that carries one or more detachable devices (as warheads). Extending the thought to enterprise architecture, it is easy to infer that the bus should be used as the transmission path or as a carrier (typically of a message) to integrate separate systems. However, since we have ESB as a product, people throw it in anytime two systems or even two services have to talk. Here are some inappropriate uses of an ESB product:

• Exposing a service using an ESB because the method signature in the EJB is not web service compatible
• Using ESB because there is a need to send a copybook, a proprietary XML or any other message format and you don't want your application to be tightly coupled to that format
• Using ESB between different layers of the application
• Using ESB because some architect decided to use canonical message format

I am not saying having ESB as a product is a bad idea but just because we have a product, we often fail to analyze the architectural need for an ESB. This leads to product first and architecture second approach, which is a bad idea. This is specially important for an ESB because it usually ends up being a central piece to which everything is connected. A bad ESB architecture will negatively impact everything that is connected to it.

In my opinion, when you have identified the need for a bus, it is often helpful to think what kind of bus you need.

• Do you need to expose business services through the bus? You need a service bus
• Do you need to send and receive messages, and in the process, need to validate, route or transform them? You need a messaging bus
• Do you have too many applications talking to one another directly resulting in an integration spaghetti? You need an integration bus
• Do you have too many sources of data in different shape, form and structure? Do you have large amounts of data moving between places but you need to improve the quality of the data before it can be delivered to the target? You need a data bus. Although the data bus is not a widely recognized concept, as an architect who comes from the middleware background, it helps me to think in terms of data bus so I can recommend a data integration solution as opposed to building services that simply move data around

Once you know what kind of bus you need, you can identify the technology that supports the bus and once the technology is known, you can select the product. This approach will lead to a more appropriate selection of the product that implements the bus pattern. There is an excellent redbook, which addresses this topic. It explains how the bus can be implemented as a pattern using different products. I can't find it anymore on ibm.com/redbooks.

In simple words, Business Activity Monitoring (BAM) deals with monitoring business solutions to extract information relevant to the business and representing it in a way that is meaningful to the business stakeholders. This may include generating appropriate notifications when relevant business events occur.

A BAM solution helps answer questions similar to the following to the business:

• What is the status of Joe's loan application?

• How many applications did Mary underwrite last month?

• What percentage of the applications were approved automatically vs. manually?

• It's October. How far are we from our goal of having 10,000 members by the end of this year?

• How do I know right away when a fraud is detected by my fraud detection system?

• How much time on average is it taking to open a new account?

Now, if you think that BAM is nothing new and business has always wanted to know how it was doing, you are right! To implement BAM, business would define 'reporting' and 'history' requirements. IT would define the data fields needed to be captured in order to provide the necessary information. IT would then write the necessary code in the application to save the information and then provide some sort of combination of SQL and web pages to view the information. This solution works very well in a typical web based application or in an application that is deployed in a single solution environment e.g. Java or .Net. From an SOA and BPM perspective, this solution is limited due to the following reasons:

• A service is a black box - unaware of the context in which it is invoked. Building the reporting logic in the service may not satisfy ALL the business contexts in which the service will be invoked. Besides, embedding monitoring logic in the services is a bad design

• A typical BPM solution enabled by SOA spans multiple IT systems and is often represented as a collection of services orchestrated in a specific sequence. Using traditional approach, logic to capture monitoring information gets scattered in the services and the information gets scattered in the databases for those services

• It is possible that business monitoring requirement would change more frequently than the core business logic. Changing application code to support changes in the monitoring requirements is ineffective from cost and time perspectives

So, what does a good monitoring solutions look like in a BPM/SOA environment?

• It can receive events in a format defined by xml schema - this way any service enabled system can emit events that can be submitted to the monitoring solution

• It can retrieve data from any relational database. This would work well when the business solution is made up of multiple services deployed in disparate systems. Different services would store their relevant data in their databases but the monitoring solution would be able to pull data from different sources

• It monitors the business solution non-intrusively. In other words, monitoring logic shouldn't have to be coded in the service implementation. Instead, monitoring solution would simply configure the deployment environment, which would generate appropriate business events when services are invoked. These events are then collected by the monitoring solution

Finally, it is a good idea to keep monitor data separate from the core business data. For example, if business wants to monitor the time it takes execute a service, don't store start_time and end_time along with the core business data that service operates on.

SOA is hot and people are curious about it. This was clear to me at IBM Impact 2008 conference when I saw more than 6000 people attending the conference. By the way, the next Impact conference - Impact 2009 - will be held in Las Vegas (again) from May 3 through May 8. I have shared my daily experiences at Impact 2008 here, here, here,  here and here.  I have described my overall experience in the following topics: 

Customer focus: There were numerous IBM customers presenting case studies and success stories. IBM seemed highly interested in answering customers' questions and getting feedback from them. As a customer, you could participate in the Birds of the Feather sessions and share your experiences, or ask questions about other customers' experiences. You could participate in feedback sessions and give direct feedback about IBM products to its representatives. You could even participate in the face to face discussions and ask specific questions about any tool or technology. I took advantage of this face to face discussions and asked questions about the future of integration between FileNet and WPS, end-to-end traceability in SOA, defining and using common elements (remember, UML is not enough in the SOA/BPM world) and the future direction around events etc. IBM even contacted me before the conference and offered to arrange a one on one session with an expert of any technology of my choice. SOA ties business and technology. Given the heterogeneity and complexity of IT and business in various organizations, the customer focus is a critical success factor for IBM or any other tool/technology/service vendor in SOA/BPM space. 

Content and coverage: The conference covered areas from web 2.0 to CICS, IT strategy to governance and everything in between. There was something... no, a lot, for everyone. Whether you wanted to learn something new or validate what you were doing, whether you wanted new insight into SOA strategy or explore ways of implementing governance, whether you wanted to see the code or pretty looking power point slides - the conference had it all.

However, I do have some critical feedback on the content and presentations. Most of the lecture style power point presentations disappointed me. The speakers were poor presenters; they knew little beyond their deck of slides and covered the topics at a very high level. The explanations like, "the message will flow from WMQBus, sendQ destination to the remote queue remoteQ of WMQ queue manager, and from there it will go to remote queue on WMQ2. Hence the bus appears as a queue manager to the WebSphere MQ and the queue manager appears as a foreign bus to the WebSphere Application Server SIBUS." don't provide any value. Most of the presentations showed screen shots of the tool, which I can see by going to an article on www.ibm.com/developerworks or simply by opening the tool. The presentations should have focused on business scenarios that warrant a particular solution, why you would pick certain solution, what are the best practices and what are the gotchas - not here is what the screen looks like. As I have explained on Day 2, labs were excellent. Labs focused on 'how-to,' the presentations should have focused on 'why.' 

I also felt that the content around business integration and BPM was too basic but having said that, I understand that the conference has to be catered to the larger audience.

World around IBM: It was great to see that many companies have products and solutions around IBM tools and technologies, which means that many vendors see IBM as the thought leader in BPM/SOA space and are willing to invest money and resources on IBM solutions. I saw vendors with solutions around hardware, software, services and even learning. 

Arrangements and logistics: I liked that every room had a dedicated TV monitor displaying events taking place in that room. A person was present outside the room and distributed feedback and forms and collected them at the end of the session. I would have liked more comfortable chairs and a table in front of every chair so people can take notes, write etc. It was little strange (and rude) when a person stopped me from entering the testing room and asked me to read the sign by the printer. The sign said there was going to be music practice in the evening so there might be some noise in the evening testing sessions. I didn't know why she had to stop me to read that sign in the afternoon but looking around little more, I realized there was another sign on the table (not next to the printer) that said no food or drink allowed in the testing center. I realized then that the person wanted me to read that sign and throw away my coffee. I thought a simple, "Sir, you can't take that coffee with you in the room." would have worked really well.

Missing pieces: I found it surprising that there wasn't much information on becoming an IBM partner. I thought given the growth potential of the SOA/BPM market, IBM would want to utilize every possible channel and every possible opportunity to get its solution to the customers and one very effective way is to use partners. It is possible that the information was available but I couldn't find the right person and location (the conference was huge).

I would have loved to see more practical information on design patterns, best practices and recommended approach around business integration, BPM and SOA - especially around WebSphere Process Server. I would also have liked some kind of solution sessions. These would be similar to the lab sessions but they would be focused on the solution, architecture and design. The sessions could present a business problem and, in a highly interactive environment, come up with the solution. They could even model the solution in the tools and leave it up to the audience to implement it.

May be a future Impact could have some design or implementation challenges. A problem or a set of problems could be posted before the conference and candidates could submit their solutions. The winners could be declared during the conference. 

Overall, the conference was a good experience for me. While the experience wasn't transformational, the conference certainly exposed me to variety of different areas and gave me new food for thought. The experience was encouraging to go next year again.

Today was the last day of Impact, at least for me, because I am not attending tomorrow. I continued attending sessions today - as always, some good, some not so much.

Kim Clark of IBM talked about WebSphere Process Server design best practices. He rather focused on problems you need to think about than how to solve them. The presentation lacked specific guidance around designing with process server. 

A technical overview on Project Zero was one of the best sessions I attended at Impact. It had less marketing and more substance. I learned the following about Project Zero:

• It's a community driven commercial project to create a development and execution platform for web 2.0 applications. It's available at projectzero.org

• You can use PHP or Groovy to develop in project zero, both of them will run on JVM. The PHP implementation has been ported to run on JVM and IBM believes most of the existing assets should run on JVM but some extensions might not

• The solution uses REST style services and all web 2.0 technologies to present the content to the user

• There is no server runtime, you just develop, zip and run

• IBM is coming up a product called WebSphere sMash (the speaker didn't give the date), which will run on top of Project Zero. Unlike other products, this will be free to download, develop and use in production. There may be some support structure but I didn't understand it quite well

Unfortunately, there wasn't enough time to show a cool demo or an application created in project zero. 

The next session I attended was SOMA: The methodology for SOA. Again, this was one of the best sessions at Impact. The speaker answered all the questions well and took the audience through a scenario that used SOMA technique to identify services. SOMA has been around for some time now and I have been using it in bits and pieces at my clients but I did learn a few new things:

• SOMA is packaged as SOMA Foundation and SOMA Complete with SOMA Agile soon to be added. Foundation is basic SOMA and Complete version is more comprehensive, useful for larger organizations and larger projects

• What I don't like about SOMA from the beginning is that you have to purchase it separately as part of Rational Method Composer. As a methodology, SOMA (and RUP for that matter) should be free. A freely available methodology makes it easy for organizations to see how something that made sense on the white board can actually be done

The third session on using WebSphere MQ and WAS together was unimpressive. The speaker didn't speak loud enough and even though I was sitting way up front, I could barely hear anything. 

In any case, the conference has ended for me. Tomorrow I will post my overall impression of the conference.