Organizations just starting with BPM often have the misconception (often unknowingly) that it's an area that allows business processes to be automated and that executing BPM activities is the responsibility of systems folks. These organizations often have business analysts that belong to systems department working with the business partners to model the business process flow. This activity is no different than the typical requirements gathering. Like they gather user-system interaction in form of use cases and business rules requirements in form of rules repository, they capture business process flow in form of process description and BPMN model. This process results in yet another artifact owned and maintained by the systems department. These artifacts are used for process automation and then, they get lost in the pile of papers that are looked up only when the system implementing the business process needs to be changed. This works well if the goal is to simply document the process flow. But BPM is not about just identifying the process flow. It is much more than that.

A successful BPM strategy requires the organizations to stop treating BPM as the requirements gathering activity owned by systems. Going back to the car analogy, business processes must be well built, owned and maintained; and this must be done by the business. IT is a stakeholder in the process of building and maintaining the business processes but the business must define it, own it and maintain it. 

This is a very important concept to develop and implement a successful BPM strategy. In my future blog entries, I will explore it in much more detail.

In simple words, Business Activity Monitoring (BAM) deals with monitoring business solutions to extract information relevant to the business and representing it in a way that is meaningful to the business stakeholders. This may include generating appropriate notifications when relevant business events occur.

A BAM solution helps answer questions similar to the following to the business:

• What is the status of Joe's loan application?

• How many applications did Mary underwrite last month?

• What percentage of the applications were approved automatically vs. manually?

• It's October. How far are we from our goal of having 10,000 members by the end of this year?

• How do I know right away when a fraud is detected by my fraud detection system?

• How much time on average is it taking to open a new account?

Now, if you think that BAM is nothing new and business has always wanted to know how it was doing, you are right! To implement BAM, business would define 'reporting' and 'history' requirements. IT would define the data fields needed to be captured in order to provide the necessary information. IT would then write the necessary code in the application to save the information and then provide some sort of combination of SQL and web pages to view the information. This solution works very well in a typical web based application or in an application that is deployed in a single solution environment e.g. Java or .Net. From an SOA and BPM perspective, this solution is limited due to the following reasons:

• A service is a black box - unaware of the context in which it is invoked. Building the reporting logic in the service may not satisfy ALL the business contexts in which the service will be invoked. Besides, embedding monitoring logic in the services is a bad design

• A typical BPM solution enabled by SOA spans multiple IT systems and is often represented as a collection of services orchestrated in a specific sequence. Using traditional approach, logic to capture monitoring information gets scattered in the services and the information gets scattered in the databases for those services

• It is possible that business monitoring requirement would change more frequently than the core business logic. Changing application code to support changes in the monitoring requirements is ineffective from cost and time perspectives

So, what does a good monitoring solutions look like in a BPM/SOA environment?

• It can receive events in a format defined by xml schema - this way any service enabled system can emit events that can be submitted to the monitoring solution

• It can retrieve data from any relational database. This would work well when the business solution is made up of multiple services deployed in disparate systems. Different services would store their relevant data in their databases but the monitoring solution would be able to pull data from different sources

• It monitors the business solution non-intrusively. In other words, monitoring logic shouldn't have to be coded in the service implementation. Instead, monitoring solution would simply configure the deployment environment, which would generate appropriate business events when services are invoked. These events are then collected by the monitoring solution

Finally, it is a good idea to keep monitor data separate from the core business data. For example, if business wants to monitor the time it takes execute a service, don't store start_time and end_time along with the core business data that service operates on.

In Business Process Modeling Notation (BPMN), an attached intermediate event is modeled as shown below. It means that the event could occur any time while the task is executing. In the example below, a Timer (or Message or Error etc.) intermediate event could go off while activity Task X is going on. If the event occurs, the process would follow its path and execute Task 2, otherwise it would execute Task 1. 

 

But I wouldn't be writing this blog if the life was so simple. While the BPMN specification isn't explicit, it does indicate that such an intermediate event results in an exception flow (see section 10.2.2 of the specification), which interrupts the normal flow. English translation of this means if the timer goes off, Task 2 will execute, Task X will terminate and Task 1 will not execute. Since the exception flow 'merges' back into the normal flow, Task 3 will execute and so on. Below, I have shown two examples with intermediate events. In the first case the behavior described above would be acceptable but in the second case it wouldn't.  

 

In the first diagram (left) if order cancellation Message is received, the Process Order task should terminate, the normal flow should terminate (Ship Order should never execute) and control should be turned over to the exception flow to execute Cancel Order task - just like it's described in the specification. 

In the second diagram (right) however, we don't want to terminate the normal flow. If the Underwrite Application task takes too long and hence interrupted by the timer intermediate event, we would like to Notify Manager but we would also like to continue to underwrite the application and eventually record the decision. If you strictly go by the specification, it doesn't support modeling the second scenario. Industry experts have been asking for non-aborting attached intermediate events.  

This strict definition of an attached intermediate event turning the normal flow into an exception flow has never been a problem for me because I chose convention over configuration. Here are some of my recommendations about attached intermediate events:

• Most of the business scenarios at my client require non-aborting attached intermediate events. So, I recommend that by default, attached intermediate events be treated as non-aborting event (I will explain the rationale in a minute)

• If we need an aborting attached intermediate event then we put a note in the model

• An attached intermediate event of type error be treated as aborting intermediate event

These recommendations work well for me because they make the models look consistent. While there is a work around in BPMN to model non-aborting intermediate events, it makes the model complex. Using convention over modeling, I keep the models simple. In most cases (at least for me) the implementation language for business models created in BPMN is BPEL, which does NOT terminate the task or normal flow when attached intermediate event occurs. Only in case of an Error event does BPEL terminates the current task and normal flow (hence the convention described in the bullet points above). So, from the implementation perspective this convention makes sense. While only a few tools generate correct BPEL out of BPMN, I have not seen any tool that generates BPEL that would terminate the normal flow in case of an intermediate event. In a nut shell, while there is a discrepancy between BPMN and BPEL specification, it's not been a problem for me so far.  

Now, about updating the specification to allow non-aborting intermediate events. The solution is simple (I hope OMG is listening). BPMN specification defines an attribute called Target for an intermediate event. Specifying a sub-process or a task as a target of an intermediate event means the event is attached to that sub-process or task. I recommend that we introduce another attribute called Abort, which would take the value True or False. The specification could force Abort attribute to be specified every time Target attribute is specified OR it could provide a default value of False for Abort attribute . The specification does do something like that today: When Trigger attribute has value of Timer, Timedate or TimeCycle attribute is required.

If you are using a tool that can generate implementation code with a terminating intermediate event, let me know. In the mean time feel free to use convention until the specification is updated.

SOA is hot and people are curious about it. This was clear to me at IBM Impact 2008 conference when I saw more than 6000 people attending the conference. By the way, the next Impact conference - Impact 2009 - will be held in Las Vegas (again) from May 3 through May 8. I have shared my daily experiences at Impact 2008 here, here, here,  here and here.  I have described my overall experience in the following topics: 

Customer focus: There were numerous IBM customers presenting case studies and success stories. IBM seemed highly interested in answering customers' questions and getting feedback from them. As a customer, you could participate in the Birds of the Feather sessions and share your experiences, or ask questions about other customers' experiences. You could participate in feedback sessions and give direct feedback about IBM products to its representatives. You could even participate in the face to face discussions and ask specific questions about any tool or technology. I took advantage of this face to face discussions and asked questions about the future of integration between FileNet and WPS, end-to-end traceability in SOA, defining and using common elements (remember, UML is not enough in the SOA/BPM world) and the future direction around events etc. IBM even contacted me before the conference and offered to arrange a one on one session with an expert of any technology of my choice. SOA ties business and technology. Given the heterogeneity and complexity of IT and business in various organizations, the customer focus is a critical success factor for IBM or any other tool/technology/service vendor in SOA/BPM space. 

Content and coverage: The conference covered areas from web 2.0 to CICS, IT strategy to governance and everything in between. There was something... no, a lot, for everyone. Whether you wanted to learn something new or validate what you were doing, whether you wanted new insight into SOA strategy or explore ways of implementing governance, whether you wanted to see the code or pretty looking power point slides - the conference had it all.

However, I do have some critical feedback on the content and presentations. Most of the lecture style power point presentations disappointed me. The speakers were poor presenters; they knew little beyond their deck of slides and covered the topics at a very high level. The explanations like, "the message will flow from WMQBus, sendQ destination to the remote queue remoteQ of WMQ queue manager, and from there it will go to remote queue on WMQ2. Hence the bus appears as a queue manager to the WebSphere MQ and the queue manager appears as a foreign bus to the WebSphere Application Server SIBUS." don't provide any value. Most of the presentations showed screen shots of the tool, which I can see by going to an article on www.ibm.com/developerworks or simply by opening the tool. The presentations should have focused on business scenarios that warrant a particular solution, why you would pick certain solution, what are the best practices and what are the gotchas - not here is what the screen looks like. As I have explained on Day 2, labs were excellent. Labs focused on 'how-to,' the presentations should have focused on 'why.' 

I also felt that the content around business integration and BPM was too basic but having said that, I understand that the conference has to be catered to the larger audience.

World around IBM: It was great to see that many companies have products and solutions around IBM tools and technologies, which means that many vendors see IBM as the thought leader in BPM/SOA space and are willing to invest money and resources on IBM solutions. I saw vendors with solutions around hardware, software, services and even learning. 

Arrangements and logistics: I liked that every room had a dedicated TV monitor displaying events taking place in that room. A person was present outside the room and distributed feedback and forms and collected them at the end of the session. I would have liked more comfortable chairs and a table in front of every chair so people can take notes, write etc. It was little strange (and rude) when a person stopped me from entering the testing room and asked me to read the sign by the printer. The sign said there was going to be music practice in the evening so there might be some noise in the evening testing sessions. I didn't know why she had to stop me to read that sign in the afternoon but looking around little more, I realized there was another sign on the table (not next to the printer) that said no food or drink allowed in the testing center. I realized then that the person wanted me to read that sign and throw away my coffee. I thought a simple, "Sir, you can't take that coffee with you in the room." would have worked really well.

Missing pieces: I found it surprising that there wasn't much information on becoming an IBM partner. I thought given the growth potential of the SOA/BPM market, IBM would want to utilize every possible channel and every possible opportunity to get its solution to the customers and one very effective way is to use partners. It is possible that the information was available but I couldn't find the right person and location (the conference was huge).

I would have loved to see more practical information on design patterns, best practices and recommended approach around business integration, BPM and SOA - especially around WebSphere Process Server. I would also have liked some kind of solution sessions. These would be similar to the lab sessions but they would be focused on the solution, architecture and design. The sessions could present a business problem and, in a highly interactive environment, come up with the solution. They could even model the solution in the tools and leave it up to the audience to implement it.

May be a future Impact could have some design or implementation challenges. A problem or a set of problems could be posted before the conference and candidates could submit their solutions. The winners could be declared during the conference. 

Overall, the conference was a good experience for me. While the experience wasn't transformational, the conference certainly exposed me to variety of different areas and gave me new food for thought. The experience was encouraging to go next year again.

Today was the last day of Impact, at least for me, because I am not attending tomorrow. I continued attending sessions today - as always, some good, some not so much.

Kim Clark of IBM talked about WebSphere Process Server design best practices. He rather focused on problems you need to think about than how to solve them. The presentation lacked specific guidance around designing with process server. 

A technical overview on Project Zero was one of the best sessions I attended at Impact. It had less marketing and more substance. I learned the following about Project Zero:

• It's a community driven commercial project to create a development and execution platform for web 2.0 applications. It's available at projectzero.org

• You can use PHP or Groovy to develop in project zero, both of them will run on JVM. The PHP implementation has been ported to run on JVM and IBM believes most of the existing assets should run on JVM but some extensions might not

• The solution uses REST style services and all web 2.0 technologies to present the content to the user

• There is no server runtime, you just develop, zip and run

• IBM is coming up a product called WebSphere sMash (the speaker didn't give the date), which will run on top of Project Zero. Unlike other products, this will be free to download, develop and use in production. There may be some support structure but I didn't understand it quite well

Unfortunately, there wasn't enough time to show a cool demo or an application created in project zero. 

The next session I attended was SOMA: The methodology for SOA. Again, this was one of the best sessions at Impact. The speaker answered all the questions well and took the audience through a scenario that used SOMA technique to identify services. SOMA has been around for some time now and I have been using it in bits and pieces at my clients but I did learn a few new things:

• SOMA is packaged as SOMA Foundation and SOMA Complete with SOMA Agile soon to be added. Foundation is basic SOMA and Complete version is more comprehensive, useful for larger organizations and larger projects

• What I don't like about SOMA from the beginning is that you have to purchase it separately as part of Rational Method Composer. As a methodology, SOMA (and RUP for that matter) should be free. A freely available methodology makes it easy for organizations to see how something that made sense on the white board can actually be done

The third session on using WebSphere MQ and WAS together was unimpressive. The speaker didn't speak loud enough and even though I was sitting way up front, I could barely hear anything. 

In any case, the conference has ended for me. Tomorrow I will post my overall impression of the conference.

On day 3 of Impact 2008, I focused quite a bit on networking but I did attend 2 sessions. 

Advanced Business Process Management Integration Techniques presentation focused on how to use WebSphere Business modeler to model a business process that partly runs to FileNet to manage documents and partly runs in WebSphere Process Server. The presentation used a lot of screenshots and the speaker went through the slides quickly leaving very little time to digest what was going on. My take away from the session can be summarized as follows:

• The integration between FileNet and process server is at primitive level, which is expected given that IBM only recently bought FileNet

• In WBM 6.1, you can export the process as XPDL format, a format recognized by FileNet

• The integration between process server and FileNet is not true integration. In other words, they don't work as one product behind the scene. FileNet exposes its process as a web service, which the process server has to invoke

• FileNet has its own framework to generate events, which are not CEI/CBE compliant events. However, WPS 6.1 comes with a FileNet event adapter, which can convert FileNet event into CBE event and log them in the events database for monitor to pickup

• The most interesting part of the example was passing attachments (documents to be stored in FileNet) between WPS and FileNet. The speaker suggested that one way was to use JAX-RPC handler, which raised several questions in my mind (none of them discussed in the presentation)
• JAX-RPC handler is part of JAX-RPC specification but from development and tooling perspective, was it always supported? In other words, did WebSphere Integration Developer provide tooling to configure and develop handlers in versions prior to 6.1? If yes, are there scenarios when it makes sense to use handlers as opposed to mediations given that handlers are lighter than mediations
• Is there any way to make attachments (or binary data) part of the BO that flows through a process? Is it a good design decision to do so? Probably not but then what is the best way to move documents through the business process whether FileNet is used or not? I am going to find answers to all these questions

The second session was [supposed to be] about using bindings in WebSphere ESB. The abstract talked about using different messaging bindings. In other words, the session was going to be about how messaging systems can connect to SCA components running in ESB, what happens to the data, how message is represented as objects etc. but the speaker announced at the start of the presentation that he was only going to talk about MQ bindings. This limited scope provided very little value. Also, during the discussion the speaker merely showed the screenshots of WID where you would go to configure bindings and set various properties. Anyone who opens WID can see those screens. The real value would have been to discuss how to set binding properties, when to select what value for the properties, when to use default selectors and binders vs. when to create custom ones. The theme of the discussion was 'here is where you configure things' as opposed to 'here is why and how to do things.'

Before the end of the day I collected some information on IBM partner program and met some friends I used to work with 5 years ago. Fortunately I didn't eat lunch at the conference so I don't have anything to rant about the food.