Thoughts on ESB

Planning SOA is a complicated process and there is a strong temptation to let a vendor offer up easy answers. Lawrence Wilkes of CBDI really drives home why it is problematic to select an ESB (Enterprise Service Bus) vendor instead of simply viewing it as an architectural pattern useful in achieving SOA goals...

"Don’t make the mistake of thinking it is safe to 'standardize' on an ESB product on an internal basis. The whole point of SOA is to provide the agility so that business and IT capability can be bestsourced. What is internal today may be external tomorrow, and vice versa.

"For these reasons," he concludes, "it is better to think of ESB as a service architecture of infrastructure capabilities, and then to consider what products best provide an implementation of each of the infrastructure services required. This may result in the selection of a product (or products) bearing the ESB label, or it may not."

Additional key thoughts from the article... (which make it a must read IMO)

• The goal of SOA is solutions based on loosely coupled, agile, federated, reusable Services. This should not just apply to the delivery of business solutions, which is rightly the organization’s prime focus, but also the implementation of the IT infrastructure. Organizations need to take care that in delivering business solutions they don’t lock themselves in infrastructure products that inhibit IT agility, and ultimately business agility.
• The capabilities required are unlikely to be delivered by a single product, nor may not be best delivery in a single product. Part of this lies in the fact that there is no common agreement on what an ESB is. ESB products often contain a mixture of service mediation, transport, management, and security capabilities. This is a useful set of capabilities to support SOA, but clearly overlaps with many other product categories. As different products in each category become better service enabled, they may be better (because of their specialization) at delivering that capability than a single “jack of all trades”.
• Federation requires the loose coupling, not just of business services and solutions, but of infrastructure. Service Providers cannot place ESB product demands on Service Consumers. Don’t make the mistake of thinking it is safe to “standardize” on an ESB product on an internal basis. The whole point of SOA is to provide the agility so that business and IT capability can be bestsourced. What is internal today may be external tomorrow, and vice versa.

Expect that line of thinking to gain momentum. At ThoughtWorks we've been advising our clients on how to resist vendor lock-in .

New Home and Articles

After many years, I finally have a personal website again. I will be posting articles there on a regular basis. The first is about Semantic Web technologies and Ruby (and a lot of other stuff) and the second is about hacking Rails for a new open-source project that I'm in the process of kicking off with Michael Granger.

Check it out at http://obiefernandez.com and let me know what you think.

SAP on Rails

The following news shouldn't be a surprise to anyone who's got their groove on with Ruby on Rails. In a new article on SAP Developer Network, german Ruby on Rails developer Piers Harding describes how to easy it is to integrate Ruby on Rails applications to SAP business-related functions via BAPI.

SAP on Rails focuses on providing an alternative type of Model template to the default - ActiveRecord. This is done via SAP::Rfc which allows RFC calls to SAP from Ruby, with automatic handling of data types, and discovery of interface definitions for RFCs. Some of this has been covered in a previous article, and on SDN. The model templating class is SAP4Rails. It's job is to automatically build up connections to R/3, look up the interface definitions for the collection of associated RFCs, and to hold the attribute definitions for the given model. This, coincidentally, corresponds to the SAP Business Objects view of the world in BAPIs.

Where's My Radar Detector

I swear someone should write a toolkit called Radar Detector that integrates your code and runs tests on a low-priority background thread while you're coding. Kind of like Eclipse does the continual compiling thing. I thought of this while reading James Shore's third great entry about continuous integration, where he says in part:

1. I often see teams using CruiseControl that don't have the shared agreement and feeling of responsibility for "we will never check in code that won't build" that underlies true continuous integration. Instead, one or two people feel that way and are using CruiseControl to force the "bozos" to step into line. You can imagine how well this works.

   If you're on an agile team and using a tool to solve a people problem, you're missing the point. "People and interactions over processes and tools," as the agile manifesto says. Get everybody to agree to take responsibility for the build, then install CruiseControl.

2. A second problem I see is teams using CruiseControl to catch their mistakes, not act as a rarely-needed safety net. This is a real and common problem--the team lets their build times get out of control, they stop running all of the tests themselves, and of course they start running into escalating build problems.

Having been on teams that displayed one or both of these anti-patterns, I understand exactly where James is coming from: Continuous integration is an attitude, not a tool and CruiseControl is a traffic cop: it doesn't help you follow the rules; it just tells you when you've broken them.

On the other hand, I've lost a lot of my idealism for being able to put together teams that maintain the right attitude and follow the rules all the time. Human nature is to follow the path of least resistance, and on a team with a continuous build server, that path is usually to check-in without thorough integration testing. Again quoting the agile manifesto, "People and interactions over processes and tools," I wonder if it isn't better to accept that people will tend to break the build and work on improving tools that prevent that from happening.

Connecting to ActiveMQ with Ruby

Oh man this rocked my socks tonight: Communicating transparently between Ruby and Java using JMS !!!

I spent the day using the web services libraries that come with Ruby to do some SOAP calls on my current project. I had some issues with getting the parameters of my SOAP envelope sorted out with the other team providing the service, but being able to code it in Ruby made it a little less stressful. I was glad to read the link above and realize I'm not the only Java guy that does this sort of thing in Ruby.

Dynamic Typing for my .NET Homies

Check this out -- Wesner Moise talks about dynamic typing and other possible changes in store for C# 3.0. I can't help but wonder if dynamic language features such as described in teh article are above the heads of both Mort and Elvis. I mean, those of us that understand the benefits of programming with a dynamic language are already using Ruby and Python. Why wait?

Deep Integration of Ruby and Semantic Web

My personal research has focused nowadays on how to best bring together two hot technologies: Semantic Web and Ruby

A significant number of people have been coming across my various blog entries about it and writing me to find out the status of my work. I have less time to work on this than I would like, but I expect to be launching an open source project within the next few weeks. In the meantime, here is a draft of my presentation that introduces the underlying technologies and key concepts. It is essentially the same version that I presented informally to various people at the 2005 International Protege Conference last week in Madrid.

Deep Integration of Ruby with Semantic Web Ontologies

Feedback welcome.

Ruby on Rails Talent

FYI: I just posted a detailed request for Ruby on Rails talent on my other blog. You want to talk about a talent crunch in a hot new technology? I think the current explosion of interest in using Ruby on Rails is just the tip of the iceberg. RUBY IS THE HOT TECHNOLOGY TO BE LEARNING RIGHT NOW!

Northrop Grumman Enters Semantic Arena

This is very important news for both Open Source and Semantic web folks! Northrop Grumman has purchased the assets of Tucana and will continue fostering development as an open-source platform.

Northrop has decided to continue development of TKS and to release a new version sometime in the future. We are still discussing feature sets, licensing and schedule, so don't bother to ask me yet. I'll post it here when I can.

The purchase also included the copyright to Kowari. Stunningly for a company their size, Northrop has not only agreed to support Kowari but rushed to do so. I certainly didn't expect a US federal systems integrator to "get" Open Source Software, but times have clearly changed. Their senior managers have made a legitimate effort to figure out the licensing and how to make it work within their business model. I have confidence that we can figure out a way to make it work for both the Kowari community and Northrop Grumman.

Via Vowel Movement

Oracle Adding Semantic Tech

I've been told that the feedback to Oracle about NDM went along the lines of "yeah that's awesome, but when are you going to support the standards?". I haven't been able to find references to this news yet, but here it is anyway: 10g.2 will include an RDF layer built on NDM -- I'm hoping this means we'll have a bulletproof and vendor-supported metastore + inference engine for building semantic web applications of the future. According to my source, Oracle will make a big marketing splash about this when 10g.2 is formally announced. Soon.

The Oracle Spatial Network Data Model (NDM) feature enables graph modeling and analysis in Oracle Database 10g. NDM explicitly stores and maintains connectivity (nodes, links, and paths) within networks and provides network analysis capability such as shortest path and connectivity analyses. NDM includes a PL/SQL API Package for Network Data Query and management, and a Java API for network creation, representation, editing and analysis.

Actually, I did find a reference to Oracle's RDF Data Model in 10g here. I'm going to try to get my hands on a copy of 10g soon and see what is provided.

Updated: It's amazing what you can find when you use the right search terms.

Enterprise Ruby on Rails

[I recently had the following email exchange with a developer who was gracious enough to allow me to repost it here, appropriately anonymized of course. His questions are mostly italicized, my responses are mostly bold. I hope this gives you an idea of what we are facing in selling Ruby on Rails as a viable technology for corporate developers. - Obie]

What in particular would make Ruby on Rails a poor choice for a large production application? Speed? Scalability? Difficulty in dealing with complexity?

Speed and scalability are not issues as far as I’m concerned. I have discussed that topic in depth with David H. Hansson and I know the details of how they have the 37signals products setup. You can create clusters with HA-proxy providing the glue between tiers and load balancers in front divvying up incoming traffic. The great thing about working with process-based platforms like Ruby is that you scale them the same as Perl CGI apps and there is a lot of wisdom on how to do that effectively thanks to years and years of practice and open-source involvement by mega-traffic sites such as LiveJournal and Slashdot.

The issue of dealing with complexity is multi-faceted. I’ve heard of some guys (via IRC) working with dozens of model objects and controllers. I think one of the biggest concerns is how to effectively share the work between more than a couple of developers working closely together -- all a matter of communication and coordinating approaches. It is so easy to do the same thing in Ruby about twenty different ways that I assume (perhaps prematurely!?) that a large team would be difficult to manage.

What would make an application too large for Ruby on Rails? Basecamp (and even Backpack) seem to be decent in size (at least in target market, and Basecamp seems significant in the functionality provided).

They have significant functionality, but the scope is very focused. That is really the whole mentality behind "Web 2.0" style apps as I understand it: do one thing and do it very, very well.

Is it possible that if an application is too "large" for this sort of technology that perhaps it is poorly architected? Yes

(Some exceptions, perhaps few?) Probably

Is it possible that many web applications are far more complex and monolithic than they should be? Most definitely!

Perhaps web applications that provide large sets of functionality (like the one I work on) should be build(sic) as smaller semi-independent modules that depend either on the other modules for interoperability or on some shared data concepts? Sounds correct

As an example, we recently developed a new module for this site. The functional requirements actually turned out to be relatively basic. However, the project took significant time to spec and build, went over budget, and is not as easily maintainable as it should be given the depth of functionality provided. Much of the additional time spent seemed (in my opinion) due to trying to shoe-horn this new module into the existing massive structure of the site. The points of necessary interaction with existing site data were few. Seems like there might be a better way to develop new functionality while still maintaining the necessary interaction with the other areas of the site as well as the availability of the data for cross-module reporting?

Well, since you use Oracle, there are a lot of people that are saying that you could design Oracle updatable views as the data sources for your Ruby on Rails apps. As for determining the suitability of Ruby on Rails, it is a productive enough environment that you could probably reimplement one or more of the modules of your web application without much effort just to prove the concept.

Here is additional commentary on the topic that I shared with him later on...

Whytheluckystiff (silly name, but famous author in Ruby-land) discusses Ruby’s memory model in more depth than has been attempted before. A small, highly-competent team of developers can be very productive with RoR and still be mindful of things like respecting memory consumption – a larger team probably not. Actually, I was thinking of my experience on large enterprise app development based on J2EE and the same holds true! So the real barriers to adoption of Ruby on Rails are probably 1) it’s new 2) the productivity factor makes it a disruptive technology in most corporate settings.

Semantics at the Roundtable

I'm participating in a series of roundtable discussions with some IT analysts and ThoughtWorkers today. Martin led off with a long discussion of language-oriented programming and we've covered a few more already. The topics are disparate but all have touched on semantics in some way -- most notably during talk about language workbenches and domain-specific languages. Those technologies facilitate intentional programming and bringing business people closer into the creative aspect of making software, but still suffer the same problems of the past when it comes to establishing shared understanding of the business domain. I'm slated to talk about Ruby but 2/3rds of my presentation is actually about Semantic Enterprise Architecture.

Semantic SOA Billing Rates

The December 2004/January 2005 issue of Business Integration Journal had an article by John Schmidt called What the &%$! Are Ontologies? in which Schmidt says, "If your consultant starts talking about semantics, metadata, cononicals [sic] and ontologies, expect to pay an extra $100 per hour."

And at the conclusion of the Protégé Short Course, the organizers passed out a menu of service offerings which included consulting on ontology development at $500 per hour. (via Semantic Arts)

Metastore Scaleability Concerns

I'm sure that I am not the only semhead concerned about scaleability issues when we start to pump millions of RDF documents into our datastore. The company that was once behind the open-source kowari metastore had a commercial offering described as...

The Tucana Knowledge Server has been developed to fill the market need for managing large quantities of RDF statements Acknowledging the problems that traditional relational database management systems (RDBMSs) have with storing large quantities of RDF data, the Tucana Knolwedge Server implements a native RDF database and consists of high-level APIs, a query engine and an underlying data store. TKS is implemented entirely in Java and is a scalable, distributed, secure, transaction-safe database built on a directed graph data model and optimized for searching metadata.

A single instance of the Knowledge Server has been used to store 350 million RDF statements and Tucana continues to improve the engine to maintain its position as the most scalable RDF data store available. Multiple instances of the Tucana Knowlege Server can be combined and treated as a "virtual database", offering another approach towards scalability. Any instance of the Knowledge Server may be used as the entry point for such a "federated" query, and will subsequently query any number of remote servers, collect their intermediate results and join on them to produce a single, coordinated result.

The problem is that Tucana seems to have gone out of business. There are myriad reasons why they might have gone out of business and I'm trying to get some information about that and whether it is still viable to base a solution on kowari. Another question I'd like to put out to the community at large is whether it makes sense to setup a hybrid architecture where low-volume data flows into the metastore, but high-volume data flows into a relational database. When we need to run a report based on a semantic query against the metastore we first load it with relevant instance data culled from the relational database and transformed back into RDF dynamically.

The side by side metastore/db idea also stands to facilitate adoption of this architecture, since it is a lot easier to sell the powers-that-be on a combined solution than it is to convince them to go from a (somewhat) unproven technology.

I had to put parentheses around unproven in the statement above because I believe that this technology is only unproven in the wide commercial arena. It is my understanding that metastores and a lot of the technologies that underlie the Semantic web are actually quite proven by medical research teams and defense contractors.

I've had this Blogger account for years and now it's time to put it to good use. JRoller has been a decent home, but the occasional bursts of crappiness finally got to me. My blog will cover topics that I find relevant in my professional life as a technologist. The subjects that excite me nowadays are dynamic languages, the semantic web, intentional and language-oriented programming. I consider these technologies to be the foundation of agile enterprise development and I'm writing a book along that line of thinking, hence the new title. - Obie