Framework as semi-finished product and component reuse

Hi,

I like to know your thoughts about the creation of frameworks and the reuse of framework components in an J2EE environmet. After reading some older posts here I got the feeling that frameworks are not the way to go.

I wonder why? Isn't a framework a good thing? I see a framework as a semi-finished product which needs some domain or customer specific extras to become the final product or application. The framework we created at our company is based on one domain model and a number of components (subjects) all referencing to the one and only one model.

The problem with using this framework come when we want to realize our customer specific solutions with it. In general you can say that we need 20% of extra domain or customer specific stuff on top of our framework to satisfy our customer requirements. However programmers don't know how to use the framework the right way. And it always ends up in a mess when it comes to reuse the framework or its components.

What programmers tend to do, is to extend the domain model and reinvent the framework components in their own project environments. Since we work in an J2EE environment our components follow J2EE. Maybe the technology forces programmers at first to take this approach. But I don't think that this is the right way our frameworks should be used and I think for another approach but don't know exactly how :?

Interestingly I also don't find solutions that tackle this problem.

What are your thoughts? Do you have some best practices or patterns one should use here.

Any hints are welcome :)

Regards,
Darya

cover of Core J2EE Patterns: Best Practices and Design Strategies, Second EditionCore J2EE Patterns: Best Practices and Design Strategies, Second Edition
author: Deepak Alur,Dan Malks,John Crupi
asin: 0131422464

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Overloaded term component

With components I mean modules containing delegates, facades, services etc. except domain objects. All domain objects live in one seperate PD model.

Regards,
Darya

Using example from Core J2EE Patterns - Order Processing System

To make the whole thing more tangible 8)

Let's discuss the topic further with an example framework. I've choosen a scenario which is described in Core J2EE Patterns at page 368 for an order processing system.

View the following class diagram. I want to use this framework in my example application.

I still need to find a little scenario for a customer specific application where I don't change the framework code but add some customer specific stuff. If you in the meantime have some scenario in mind feel free and adapt the application with above framework ;).

Regards,
Darya

A Scenario

Now, having the framework visualized I'm going to use it. There are multiple scenarios thinkable and I start with a first one.

Think about you want to use the framework in your specific application. The only difference is that in your application the Order class needs an addidtional attribute let's say for a priority.

Think about that you can not simply extend the framework Order class and add priority, because that would mean that you can't use the component anymore.

What would be a elegant solution to it. The goal is to reuse the component based framework at best.

Any ideas :) ?

Regards,
Darya

Jeff De Luca's picture

Frameworks - Reuse

Hi Darya,

there's several questions mixed in this thread. I'm going to tackle the high-level issue of frameworks in general.

This is not new thinking. In fact, I'm going to paste in (most of) the content of an e-mail reply I wrote on this in September 1999.

The biggest issue with frameworks is that of omniscient design. That is they somehow need to anticipate the requirements of each and every domain and each and every specific project within a domain, each time they are used. This of course is quite impossible and any attempts to be so truly generic and pervasive, result in a complex mess. Hence, framework implementors need to make assumptions and set boundaries of use. Even in a very specific, narrow domain (such as a collection framework) there are dozens and dozens of such assumptions made. Consider the very different approaches used by the Java JDK1.2 collections package and the JGL collections package; and this is a very narrow domain.

There is an excellent chapter on JavaBeans in the Core Java book. Here, for just a single component, the difficulties of trying to design and implement something so generic are laid bare. Now, extrapolate this from a single component, through
to a narrow domain class library (such as a collection framework) and then to something with the breadth of the San Francisco framework...

Not only are there such fundamental design and boundary assumptions made there are also a multitude of decisions that affect syntax and semantic usage. For example, the naming and ordering of parameters, whether returns codes or
exceptions are used, the granularity of exceptions used, and so on. This is especially problematic when using class libraries from multiple different vendors. Say a collection framework, a printing framework and a 2D graphics framework. (note: I am ignoring the even more problematic issue of thread safety and thread design and implementation as well as the issue of inheritance use by a framework - you only get to inherit once in Java...).

Now, if your particular needs and means of usage happen to intersect with the assumptions (design points) of the framework vendor then you can gain quite a bit, but this is the key point: In essence, to use a framework means you must work to its design points, its set of assumptions, its syntactic and semantic usage and so on. To continually try and fight against these issues in any particular framework undermines the value proposition of the framework in the first place (after all, you chose that route to get significant reuse).

If the framework seems too rigid or inflexible and you are having difficulty mapping against it then, in the absence of other data from you, this suggests the fit of this framework is not good. You can drop or replace the framework or drastically alter your requirements of it.

The next level of this discussion I've briefly described here (within a different context). link

Framework Reuse - Product Line Development - Lead Time Reduction

Hi Jeff,

object reuse, framework reuse, component reuse or any kind of reuse are good as long as they beware us from reinventing the wheel. When reading your comments on framework reuse I am more encouraged to give up the idea of creating own frameworks.

Think you are in one and the same business and that your problem domain always look similar. You customer believe in your competence and gives you the job to create a similar solution they saw from you at a partner site. After a while and a number of projects you recognize that you could drastically reduce your lead time when you had a platform in place which you only need to customize so that each new customer benefits from short delivery times. Or even go so far and let do the customization from the customer's IT department.

A framework steps in right here. I really don't see why it should not work. We are talking about PD models, but there is more than that. If we want to create PD models in a really reusable fashion than it is the framework that force us to think about how to implement these PD models. Look at the Order class in my previous diagram. The priority attribute is missing to implement my new feature. So I need my own Order to extend the Order class.

What about the factory pattern? Wouldn't a factory in my OrderAppService class help here? Instead of new Order() I would say createOrder(type). And if I had a whole PD object graph to implement, wouldn't the OrderAppService be the right place for a plug-in point. So that my custom layer would have to implement the object graph.

I really like to hear more thoughts from you, if possible with references to the given order processing system example.

Regards,
Darya

Jeff De Luca's picture

Frameworks, Reuse, PD layer, et al

Darya,

it's hard for me to know exactly where you're pitching this. I'm not entirely sure we have the same meaning for framework in mind.

darya wrote:
Hi Jeff,

object reuse, framework reuse, component reuse or any kind of reuse are good

They can be good where the cost of doing them is ok and they can actually be reused effectively. That is, without one having to fight against the thing to be reused or to try and make it fit into a different set of semantics. Reuse for the sake of reuse is not necessarily a good thing.

There are all kinds of reuse. For example, we can reuse knowledge.

darya wrote:
as long as they beware us from reinventing the wheel. When reading your comments on framework reuse I am more encouraged to give up the idea of creating own frameworks.

No, don't rush to that judgement.

darya wrote:
Think you are in one and the same business and that your problem domain always look similar. You customer believe in your competence and gives you the job to create a similar solution they saw from you at a partner site. After a while and a number of projects you recognize that you could drastically reduce your lead time when you had a platform in place which you only need to customize so that each new customer benefits from short delivery times. Or even go so far and let do the customization from the customer's IT department.

"You" is a real problem word in that paragraph. When you say "you" - who is that person? The first several times I read it I thought "you" meant me. Now I think "you" means anyone. What I write now assumes "you" means any person in general.

(side note: this is a great example of "you" as a disconnect in the straight-talk section of the workshop. look back on those slides sometime)

There's a lot of ways to reduce the time it takes to do something. There's an implicit assumption in what you're saying that reusing code is the only way to do that. What percentage of the SDLC is coding?

One can certainly reuse knowledge in a case such as you describe - and that does not even have to mean domain specific knowledge. Let's say a project manager hires some really talented people to help him on some project. These people are experienced color modelers and have significant application design and development experience. A-class developers as I call them.

The injection of those people, and the knowledge they will reuse, will have a dramatic impact on that project. And these people may not even have any domain specific knowledge. They know how to model. They know how to design. They know how to code. There's all sorts of reuse happening there - and they will produce faster because of it. But they are not necessarily reusing previous lines of code, line by line.

Don't get me wrong. I'm not saying never reuse code. Right now, I'm just trying to better understand what you really mean and to try and get you thinking more broadly.

darya wrote:
A framework steps in right here. I really don't see why it should not work. We are talking about PD models, but there is more than that. If we want to create PD models in a really reusable fashion than it is the framework that force us to think about how to implement these PD models. Look at the Order class in my previous diagram. The priority attribute is missing to implement my new feature. So I need my own Order to extend the Order class.

Ok, here I think is the big disconnect. What we talk about when we are modeling the PD for some project is the PD layer of a single application. We are not talking about some enterprise object model. The model is for the PD layer of a single app. It is how the PD code of a single app will be structured.

Here's a diagram of the application architecture.

Application Architecture - Simple

When I say we are modeling the PD for a project we are modeling that PD layer (business logic layer) for that one app on the left (for example).

I'm not talking about some enterprise object model or even wider.

I understand the desire to go down that road. After all it is a somewhat compelling and seductive notion. And as an industry, we have been there before. Many times. And it just hasn't worked out.

When data modeling first came along many thought this is great. It is so great that we could create the one big data model for everything that everyone could just reuse. And then it was realised that, well, the whole financial industry is really something of its own (just to pick one example here) and it's sufficiently different from materials resource management that they really need their own data models. And then it was realised that the financial industry data model was problematic as banking really is different from insurance and so there could be one banking industry data model and one insurance industry data model. And then it was realised that, well, really banking in the USA was pretty different from banking in Australia. In the USA you had lots of small community banks and in Australia you had a tiny number of very big banks with big branch networks. Some did debit cards, some don't, and so on.

At this point, as an industry, the notion of such data models went away (yes I'm generalising here - there are still pockets today that believe in such things). But the notion and example can be carried forward. There could be an Australian banking data model. Ah, but AusBank1 has a strategy of lowering the cost per transaction. AusBank2 has a strategy of increasing share of certain types of lending.

Now, even if we just limit the example to lending. Then sure, within lending, the statutory and regulatory checks for lending will be the same at AusBank1 and AusBank2 (as the stat-reg body is external to both of them) but their own internal credit policies will be radically different. Not only is it that the checks are different, it is also the product types and connections and so on as one bank is entirely excluding such things where the other is explicitly going after them.

Now, this didn't work for data models. We can't go to the I.T. retail outlet down the road and buy the MRP data model (for example) and drop that in and start using it.

But we tried again when objects came along. But with objects, it is much much harder because we are not just trying to describe common data. Objects contain behaviour as well as data. So, where we couldn't get a data model to stand up across organisations, with an object model we need common data and behaviour and its structuring to stand up across organisations.

At the framework level Taligent (IBM and Apple) tried in terms of OS and app development frameworks. IBM tried again with San Francisco in terms of line of business frameworks.

At the other extreme we have discrete objects (classes). It was thought we'd be able to run down to the I.T. retail outlet and buy ourselves a Customer object - ready to drop in and reuse. The same thing again with JavaBeans and then of course with EJBs (and let's just say that the "for sale" reusable bean industry is not exactly experiencing stellar growth).

Customer is often thought of as potential enterprise class. Well is it? Let's say our job is to design and build a Customer class for general reuse. What goes in it? Well, if the domain is broad this is an impossible problem. If the domain is Point-Of-Sale (POS) Cash Sales - then Customer is a class that is not even needed in that domain. At the other extreme is a domain something like a mega-CRM - then Customer in that domain is a big thing with lots of biographics in it and demographic links and so on.

Where in that continuum should we design and build our generally reusable Customer?

We can't have one Customer that satisfies all - to do so means it must be at the big extreme end which then means it is cumbersome, unwieldy and overweight for just about everyone else.

So, this kind of thing is only achievable if the domain is sufficiently narrow.

Given a narrow domain, can we design for flexibility. Of course we can and it's one of the big tradeoff decisions that are being made all the time. Peter Coad used the analogy of printed circuit boards and Java interfaces (as just one example of this). On a PCB the components can be hardwired. It's very simple and easy to understand, but it's not easy to change. A PCB can also use a socket where flexibility is needed. But PCBs aren't designed where everything is sockets. It's too costly and too complex. Similarly, a Java interface can be used for a plugin point for algorithmic flexibility, but we don't design everything as interfaces.

There are many other similar design for flexibility approaches. Here's a simple FDD model of the structure of the FBS (features list).

An FDD Feature

It's simple. It's easy to understand and it fits the FDD way of doing an FBS well.

If I wanted flexibility in this structure - for example, so that people can have any number of levels within it then I might model that like this.

FBSNode

A person, can reuse their knowledge of analysis, design and implementation and design and code for flexibility where required. They're reusing approaches, patterns, and so on.

Now, finally we get to reusing code. Sure that is done lots of times. UI is an easy example. You might design and build your own calendar UI component or you might buy and reuse one.

Notice how very specific the domains are now.

Ok, so switching back to root of this. You asked a question about frameworks and plenty of I.T. folk will jump in with "here's how I do one" or similar. I'm more interested in what your real problem is. I see three topics here - (1) frameworks, (2) lead time (to develop/deliver), and (3) designing for flexibility. I suspect framework is an assumption about the solution to 2 or 3 or both (?)

If the problem (and thus question) were stated differently we'd be having a different discussion here.

darya wrote:
What about the factory pattern? Wouldn't a factory in my OrderAppService class help here? Instead of new Order() I would say createOrder(type). And if I had a whole PD object graph to implement, wouldn't the OrderAppService be the right place for a plug-in point. So that my custom layer would have to implement the object graph.

I really like to hear more thoughts from you, if possible with references to the given order processing system example.

Regards,
Darya

I don't think we have anything to add beyond what's already been discussed in this thread. We just wouldn't do things that way.

But this part is really a separate discussion from that above (this is topic 4).

Another point of view

Hi Jeff, long time no post.

I agree with Jeff entirely Daria, but I would'nt go as far as to believe that Frameworks are not the way to go as you seem to conclude. I think they are, more and more. Today a programmers don't do much thing without frameworks in general. As for "Business Frameworks Oriented" well just the same thing, SAP, Microsoft Dynamics, SIEBEL, PeopleSoft, Cogen just to name a few are Solutions/Frameworks IMHO and they are being used more and more.

I agree with Jeff that Class Frameworks such as the one you presented are not the way but you cannot extrapolate from that the Frameworks don't work in most cases. Consider Frameworks that have their own programming language, development tools, designers and code generators. We have now DSL tools that generate "controlled" code according to a predefined schema and rules, etc etc, the technology is moving forward fast. Some tools allow one to work an define what I call Meta-Information (or Meta Classes). That is information that you define until a concrete class is still to be realized in the system. Realization occurs only when code is generated.

The question is, can most companies afford them finantially for all cases? Well, no, that is why custom development will not go away that soon.

"What programmers tend to do, is to extend the domain model and reinvent the framework components in their own project environments. Since we work in an J2EE environment our components follow J2EE. Maybe the technology forces programmers at first to take this approach. "

Yes, this is not the way to go. But imagine that you can extend the domain model using some DSL tools and framework components are generated affter accordingly (code). This is being done today with success, the problem is ... most hard core programers don't like it. Why? Technically I don't know, culturaly I can see why.

Having said this, Frameworks are not a silver bullet. There will be always something they don't do properly or that they can't do. The question is, that this invalidate its usage?

Today is more important to a person to say I know how to work with framework X or Y and this is my experience, then to say I know Java, Pythong, Perls or whatever and this is my experience, IMHO.

Programming became a commodity, but still today good programmers/architects are hard to find.

Nuno
PS: The bottom line Frameworks speed up the development process, but in the other hand became technically more important to master them then a programming language or programming technique.

An example of mine

I'm working with a top Insurance company in Portugal. The company uses a lor of frameworks and development environments and technology. SAP, Cogen, Net Dynamics, WebMethods and so many others. We also have a lot of home made frameworks. I currently work with a User Interface team. This team is reponsible for 90% of User Interface development the occurs in the company. The UI is totally Web Driven. The common scenario is that the UI calls the EAI (WebMethods) and then the second calls whatever systems necessary to fetch and commit business information. We use a SOA architecture.

In the past the team would receive some scratchs of forms designed in PowerPoint, then coded by hand HTML Pages, Web Components etc. We have built a Class Framework to deal with user interaction flow, state management, calling web-services etc etc. A complex form would take 5 days to code, another 2 to test it. The Framework was good and extensible. We are always extending the Framework to include new functionality, new interaction behaviours and logic. We came to realize that in order to turn more reusable and reduce the development another approach a Class Framework was just not enough.

So we designed a code generator (cross compiler), Visio stencils that allows anyone to define the user interaction processes, forms etc using Visio. So now the team gets say all the Forms and User Input Processes in Visio Diagrams. Then we execute the cross compiler that transforms the data in visions to code in our framework. Developer do not toutch any HTML code, only code behind. But even there most is done, they only need to code logic to disable and enable controls according to state, or to decide what wiil be the next step in te user interaction process. Something that would take 5 days to code, takes now 1 day. Changes in the diagrams propagate to code, but not vice versa. Any change in the UI visuals or Flow need to be done in Visio. We have managed to standartize input formats across all UIs of all applications etc.

So is the end code resusable? Yes it is, we often use the same process and forms in multiple contexts in the company. Yes we need to adapt it here and there but we do not build the same screen from scratch just becouse we need on more field here and there, or even if the behaviour somewhat changes.

How does it all fit with FDD. Perfectly :)

Our Problem Domain is always within the scope of UI. Class diagrams where not good to this kind of things so we developed our own DSL and tools. The Visio Stencils and Cross Compiler. We also have Class Framework that all sorts of things, let's call it our extensible engine. This engine gets extended by the code generated by the cross compiler and by programmers. In sum:

* Build an Overall Domain Model - Done in Visio. The frameworks even generated a "live" prototype of the UI. We don't use Class Diagrams or DNC
* Build a Feature List - Feature list are defined by reading the UI Interaction Processes and Form Transitions. A UI Interation might be Fiding a Customer, Selecting a Contract for a Customer etc.
* Plan By Feature - Business as usual
* Design By Feature - Developers update the visios wirh the webservices to be called for UI Interaction Process. This is mostly done already by system analysts.
* Build By Feature - Business as usual

Could this be achieved with UML and Class Diagrams? Yes, but by using a DSL created by us we have managed to simplify the UI Modeling Process and generate code from those model more efficientely then any other approach we tried. The problem was really in UML and OO concepts in it and the notion that everything can be done efficiently and universilly with it. Think out of the box.

Nuno Lopes

Jeff De Luca's picture

Welcome Back

nbplopes wrote:
Hi Jeff, long time no post.

Hi Nuno,

welcome back!

Smiling

There is No Requirement for Domain Specific Reuse

Darya, I'm going to come at this from a different direction and conclude organizations and their systems don't want or need domain specific reuse.

What organizations do is highly specific (to the organization), although organizations in similar businesses may operate in similar ways and some things like ordering (say office supplies) is similar across many organizations.

Most organizations do every type of thing they need to do in just one way, and by extension have just one information system to do it. And if they have more than one system doing the same thing, they are anxious to eliminate the duplication.

Within the organization, systems (processes) are generally organized by the domain. By this I mean, one part of the organization does/processes some types of interactions and another part does different types of interactions. In some ways this is an easy concept to get across – orders are processed in the order processing department; the lending department processes loans – but has important consequences for reuse.

I should explain that an interaction is anything an organization does with anybody or thing outside itself (or with another part of the organization – note this qualification needs elaboration). Organizations exist to perform these interactions and arguably are an organization's only reason to exist and maintaining a record of them is the primary purpose of its systems.

To me, an interaction is a Moment-Interval (but may be multiple M-Is), but Jeff and others may not agree with this.

It's easy to envisage components that many organizations and their systems could (re)use, such as an emailer component (it happens that I have built and sold these kinds of components), but as soon as you start to make components specific to a particular organization and what it does, reuse gets a lot harder. Why is this?

Putting aside issues of whether the framework components accurately model the organization (if you like it's domain).

It's because organizations with few exceptions organize themselves along the lines of their domain and one part of the domain (i.e. one type of interaction) is done in one place and in one way, the organization has no need for reuse in its domain. Note, that it could potentially reuse (once only) a domain specific component from outside the organization, although with the issues Jeff details.

But, I hear you say, almost all of our systems have Customer in them. Why can't I have a customer component that I can reuse?

As I said, organizations are structured along the lines of the interactions they process and so should their systems (in fact they have to). Typically, there are many different types of interaction with a Customer. So different parts of the organization are recording different things about the Customer. I know one part of the organization often has to communicate with another to get information or to get them to do something, e.g. the credit Card Department tells the Current Account Department to debit a customer's account for a credit card payment. But these should be (I'd say, must be) handled by interfaces between systems (or within systems), not by reuse.

I think the mistake people make, is to think of the domain in terms of the things in it, when they need to think of the domain in terms of the interactions the organization performs. Doesn't sound very OO, does it? Smiling In design terms, it means you have a customer class/component specific to that particular part of the domain everywhere (in every system) you need one and that class/component presents interfaces to provide information and services to whoever else needs them.

BTW, an interaction is not a process, although it is invariably handled by a process. I tend to think of an interaction as being like an interface, that could be implemented by several different implementations (i.e. processes).

Best regards

Phil