AZ Givecamp for Non-Profit Organizations

Hello Everyone,

AZGiveCamp is organizing an event for developing Software Applications for Non-Profit/Charity Organizations on the weekend of May 21st 2010. It's a event where group of software engineers team up and develop applications for Non-Profit companies that they desperately need to improve efficiencies, streamline their business processes so that they could reach out to their users/customers and serve their communities better.

The kind of applications buing built range from getting a simple website up and running, improving the existing system architecture to make it more effiecient, make the website look more prettier than the existing one, develop an application from scratch to stream line their business process etc.,

There are wide variety of companies that have already registered and the kind of applications they are looking for range from simple to fairly complex. So, depending upon your skillset and the amount of time you are willing to spend, you can choose what project you are interested to work on.

The organizers are looking for Dev Leads, PM's, BA's and Software Developers. If you are interested or know of anyone who can contribute to this camp, please signup on the AZGiveCamp website at http://azgivecamp.org/Home.aspx.

Thank you and please pass on if you know of anyone who is interested.

--
Preetham Reddy
http://www.preetham-reddy.com/

Must have skills for a Software Engineer!!!

Ever wondered what does it take to be a successful and highly productive software developer??? Well, The success lies in not just being good at what you do but having a fair knowledge across wide spectrum of technologies and constantly learning new stuff and applying those things in your day to day life to improve the efficiency of applications you write.

Somebody once said - "A programmer who hasn't been exposed to all four of the imperative, functional, objective, and logical programming styles has one or more conceptual blindspots. It's like knowing how to boil but not fry".

He was spot on. There are lot of technologies/methodologies involved in the design and development of Application Software. Quite frankly, each one of those software development methodologies/technologies/languages are designed to solve a problem situation in a better way than other ones. There is no one size fits all. It all depends on the problem at hand, developer expertise, budget and most importantly, the available time frame to deliver the project.

Here's what I think are the must have skills :

• Expertise in Object Oriented Programming languages like C#, Java, VB.NET along with their associated frameworks like Asp.net/ J2EE.
• Legacy languages like C, C++; C++ is mother of all modern day programming languages like C# and Java, which inturn was derived from C. If you are new to software development, I would suggest you to start learning languages in the following order, provided you have time. C --> C++ --> C#/Java. The reason being C# and Java borrow lot of concepts from C++. Thorough understanding of these legacy languages will only strengthen your understanding of C#/Java and you can borrow most of the coding style etc/.,
• Data Modeling Languages like UML and their associated tools like Rational Rose or Microsoft Visio. Before you get down to writing code, you have to design the application and put it down on paper using various models for pictorial representation of your application architecture. Unified Modeling Language is a must have skill.
• Relational Databases : Sql Server and Oracle are the most widely used databases for Enterprise Applications. You've got to know how to Design/Architect database tables, the relation between them, apply constraints and also to normalize them to maintain integrity and reduce redundancy.
• Programming T-SQL/PL-SQL : Database is an integral part of majority of the applications. To be a good developer/architect, you not only need to have a very good understanding of relational databases but also how to program them. You've got to know how to perform various operations, write complex queries to process data, apply business rules etc., And for that, you've got to master the art of Sql Stored Procedures.,
• ORM Technologies : Object Relational Mapping technologies like NHibernate, Entity Framework, Linq To SQL or LLBNGen Pro helps you in developing your database intensive applications rapidly by abstractiong out most of the plumbing code to access your database objects. Each one of the technologies mentioned above have their pro's and con's. Some of them are open source and other's are licensed. Depending on your need and the kind of features you are looking for, one of them can fit your needs. My favourite in NHibernate.
• Third Party Frameworks : Depending upon the complexity of your application, you might need to leverage some of the benifits of third party frameworks to help your build quality applications. One of my favourite is Spring.NET which a port of Java's Spring Framework. Spring.NET is lot of things for lot of people. It has lot of components that aim to improve the speed and efficiency of your applications. My favourite feature is Dependency Injection (Inversion of Control). There are other IOC frameworks too. Once you get the hand of them, you'll never go back to your normal style.
• CSS : Cascading Sytle Sheets for the look and feel of your website.
• JavaScript Libraries : JQuery, Scriptalacious, Prototype and YUI are the most popular ones. My favourite ones are YUI (Yahoo User Interface) and jQuery.
• Ajile Development Methodologies : Scrum / Kanban using TDD ( Test Driven Development).
• Mobile Phone : Let's face it. Mobile PDA's are the future of internet. You don't want to be left behind in area. With Monotouch, you could develop iPhone/iPad applications using C#.
• EAI : Enterprise Application Integration Tools like BizTalk / Business Objects. A good software engineer knows how to leverage best tools in the market to integrate applications developed in different platforms. Although you can make use of MSMQ and Windows Work Flows to perform most of the tasks needed for application integration, using tools like BizTalk provide you with support for Load Balancing, Applying Transformations, Dynamic Routing that are required for large scale applications.
• RIA : Rich Internet Applications are the order of the day. If your web-applications are target at consumers, you've got to master atleast one of the following technologies : Silverlight/WPF/Flash.
• Azure/oData/Oslo anyone????

Although the above list looks very long, in-reality its not. Most good developers have a fair knowledge of most of the technologies mentioned above and I don't see any reason why you should not!!!

Case for ESB Toolkit 2.0 : Why do we need it?

It's been over 6 months now, since I've started digging into ESB Guidance 2.0 CTP version ( and later ESB Toolkit) and working on an 'Enterprise Service Bus' solution for an Insurance company that would like to have it's systems integrate seamlessly with various 'Service Providers' on the fly, in a very secure, dynamic, reliable way and yet being loosely coupled.

The resulting system should give the end users, an ablilty to configure various applications to consume services offered by a large array of 'Service Providers'. They should also be able to track the messages that go in and out of the hub, do exception management, fault tracking and generate various reports to make business decisions on the fly, all from a web interface. We've been going back and forth, trying out various technologies to find out the best way to implement such a system. After an exhaustive and a very extensive analysis of various ESB products, we finally settled on 'ESB Toolkit 2.0' and 'BizTalk Server 2009'. The application that we developed using ESB Toolkit looks very elegant and provides solution for every single bit of problem we've set out to solve.

'ESB Toolkit' is Microsoft's flavour of 'Enterprise Service Bus' implemented on BizTalk Server 2009 platform. There are plethora of third party vendors who came up with ESB products to help enterprises design and implement their EAI projects. With ESB Tookit 2.0, Microsoft plans to capture a pie of that market. Before we discuss 'ESB Toolkit', we've got to first understand what is 'Enterprise Service Bus'? What role does it play in integrating applications? What solutions does it provide that traditional EAI integrating solutions doesn't?

'Enterprise Service Bus' is an architectural pattern that enables the development of 'Service Oriented Architectural' applications. That begs the question, What is 'Service Oriented Architecture'? While there is lot of depth to the concept, SOA at a very highlevel could be described as an architectural pattern that will allow you to package functionality in your software as a set of interoperable services in a loosely coupled way. It adds an additional layer to your application that consists of set of services that are provided by it. These services are exposed to consumers and they can use one or more of these services to implement business logic in their programs. For any application to fully implement SOA, it has to support these features :

• Services should be loosely coupled with Operating Systems. Consumers had Providers should not be depended on each other or the operation system they are developed on to communicate with each other.
• Services should be flexible. They need to be accessible through variety of transport protocols.
• Services should be agile. Any change in business requirements should be quickly propagated and deployed without much hassle.
• Services should be re-usable. They should be easily composed into new business processes.

It is possible to design solutions that consume these services whenever and in whatever way, without the use of an ESB. However, such an implementation has many drawbacks, which would be discussed later in the blog. 'Enterprise Service Bus' is an additional layer(middle-tier infrastructure) added to applications that would help their clients to consume their services in a structured way by providing a wide array of tools. ESB is just one building block of SOA applications. Since ESB assists in the development of SOA applications, it has to align itself to the core principles of SOA namely loose coupling, flexibilty, re-usablilty and agility etc., apart from providing the following features.

• Dynamic Resolution of endpoints for Routing.
• Content based Routing.
• Routing based on Rules Engine. ( In case of BizTalk, it's BRE).
• Message Transformation using Maps ( In case of BizTalk, it's BizTalk Mapper).
• Message Enrichment.
• Service Aggregation.
• Support for multiple transport protocols ( Http, Ftp, MSMQ, FILE etc.,)
• Business Process Orchestration.
• Business Activity Monitoring.
• Message Logging.
• Reports.
• Alerts.
• Exception Managment.
• Support for Service Registries eg UDDI.
• Itineraries(Mediation Policies), which consists of instructions to ESB regarding what components to use in the underlying engine, to process the message.

What does BizTalk provide 'out of the box', what it lacks and how does the ESB Tookit fills the gap???

BizTalk is primarily designed to be an 'Enterprise Application Integration' (EAI) tool. It's primary function is to integrate various applications (Service Providers and Service Consumers) using a 'Hub and Spoke' model. BizTalk acts as a hub which accepts an inbound message and depending upon the need, performs various operations like transformation, enrichment, routing etc., before forwarding it to appropriate subscribers. If it's a two way request, it waits for the subscriber to return the message, perform another set of outbound operations before sending it back to the consumer application. It achieves this functionality by providing a series of 'Receive Ports', 'Receive Locations' and 'Send Ports'. To be able to consume a service, the consumer application should have an endpoint where it can send the request. For this, a BizTalk developer typically provides a 'Receive Port' and a 'Receive Location' which consists of the required endpoint. This receive location will have various 'Pipelines', 'Maps' and other BizTalk components associated with it to perform the necessary inbound operations like 'Transformation', 'Decoding', 'Enrichment' etc., Once these operations are performed, it will be forwarded to the subscribing send ports. It's the job of the send port to subscribe to the messages arriving on the receive ports and based on the context, forward it to appropriate service provider. Send port will have all the necessary details like URL, transport protocol etc., It has it's own set of pipelines, maps associated with it to perform the necessary operations before sending the request and after getting the response. When it finally get the response, it forwards it back to the same receive port through which the response message is sent to the caller.

The problem with this approach is, it gets overwhelming after a certain point. BizTalk should know about every single message that it would process. This means that the corresponding BizTalk application should have schemas for all the inbound and all the outbound messages. It receive ports should be configured to pickup specific messages from exactly one location. Send ports should be configured to specific locations. All these configurations should be done during the design time. All the operations that needs to be performed on inbound and outbound messages should be specified during design time too. If the application ever gets too large and complex, it would be very difficult to sustain this kind of architecture. The BizTalk Server would be flooded with large number of 'Receive Ports', 'Receive Locations', 'Send Ports' that gets very difficult to manage and maintain. This architecture is very tightly coupled and neither flexible nor agile.

While there are lot of ways to provide more flexibility, agilty and interoperabililty to BizTalk applications, it requires lot of work on the part of programmer and the solution isn't normally elegant.

This is where 'ESB Toolkit' comes into the picture and fills all the required gaps!!!

ESB Toolkit 2.0 using BizTalk Server 2009 :

ESB Tookit extends the functionality of 'BizTalk Server 2009' by providing wide range of capabilities to it that didn't exist before. It provides a bunch of tools to developers that they could leverage to build robust, connected, service-oriented applications that incorporate itinerary-based service invocation for lightweight service composition and dynamic resolution of endpoints and maps. They would also help in Web service and WS-* integration, fault management, exception management, business activity monitoring and reporting.

'ESB Toolkit' cannot exist without the BizTalk Server. It is built on top of BizTalk and makes use of various components like 'Pipelines', 'Maps', 'Orchestrations' that come along with BizTalk.

ESB Toolkit solves the above mentioned problem of schemas by releasing the need to have schemas for any of it's mediation components like on-ramp, off-ramp, generic routing and generic transformation agents. The ESB solution should be generic, re-usable, flexible and agile. For that to happen, the various components provided by ESB Toolkit should not be tied down to schemas.

For example, a typical BizTalk solution for consuming a webservice would be something like this. A schema would be exposed as a webservice by creating a virtual directory in IIS and creating the necessary receive ports and receive locations. Then, a send port is created and configured to subscribe to that receive location and send a WCF request to the webservice. For every webservice that needs to be consumed, you've got to create seperate receive location and a send port. Now, if the WSDL of any of the already configured webservices should change, then you've got to open the VS, import the new WSDL, compile the project, deploy it and restart the host instance. This is time consuming and ridiculous way of implementing SOA solution.

If the need to have an XML for the incoming and outgoing messages is removed, then it makes life a lot easier for developers and the resulting solutions would look more elegant. With ESB Toolkit, the OnRamp and OffRamp would be binded to generic/dynamic ports which does not have to validate the incomming message against a schema. Since the webservices already do the validation, it doesn't make sense to add another layer of validation in BizTalk. The only requirment is that the messages should be well formed. They should be valid XML documents. This is the pattern followed by all the Itinerary(Mediation) components in ESB Toolkit. They all have inbuilt support for 'Generic XML' so that they don't need to change when webservices change their schema or WSDL.

ESB Toolkit comes with a bunch of webservices which helps in dynamically resolving endpoints, pullout required transformation files (maps) and process mediation policies. It also provides an 'ESB Management Console' Portal which gives various reports on Faults, Exceptions etc.,. It allows business users to have a look at the fault messages, it's context properties and gives them an ability to change the message and resubmit it. The users would also be able to subscribe to alerts, configure registries and finally trace the message as it goes in and out of the BizTalk Server.

Therefore, ESB Toolkit 2.0 is a very welcome addition to BizTalk Server 2009, and it significantly increases the functionality and capabilities than that were previously offered by BizTalk. This also ensures that the BizTalk server is no longer thought of just as an EAI Toolkit but a very important engine in the design and development of SOA applications.

In my next series of posts , I am going to dig deeper into the ESB Toolkit Architecture and various components that ship with it and how to leverage those components to develop loosely coupled applications.--

Preetham Reddy Chamakura

www.preetham-reddy.com

Introduction to ESB Toolkit 2.0 Itineraries : Video

Hello Folks,

This is my first video of the ESB Toolkit 2.0 video series which explains the basic concepts of an Itinerary. In this video, I create a sample Itinerary that makes use of existing BizTalk components like Maps and Orchestration.

It takes in an Input Message, Transforms it using a BizTalk Map, then uses the Orchestration Extender to route the message to a WebService. Once it gets the response, it sends it back to the caller.

In my subsequent videos, I am going to walk you through creating more advanced Orchestrations, Maps etc., and leverage those in an Itinerary. Until then, enjoy this video.

Here is the link to the video : Introduction to ESB Toolkit 2.0 Itineraries

Btw, Today is the 63rd Independence day of my favourite country in this world, 'India'. Happy Independence day to all the Indian's out there!!!

Cheers,
--
Preetham Reddy Chamakura

www.preetham-reddy.com

PS : Please turn on your volume full throttle... There is some problem with the way my software recorded my voice... I'll rectify it in my following videos!!!

ESB Toolkit : Custom Orchestration not showing in the Itinerary Property Window

Ever tried to use a custom 'Orchestration' in your Itinerary for processing your business funtionality only to find out that 'Itinerary Designer' is not showing it in the 'Itinerary Service' properties??

Well, there are couple of things you need to do, before you get to that point!!!

First of all, You need to create a new entry for your 'Orchestration' in the ESB.Config file located at 'C:\Program Files\Microsoft BizTalk ESB Toolkit 2.0\Esb.config'. Open up the configuration file, look for 'itineraryServices' node. Inside this, you've got to add a new entry for an 'ItineraryService' of type Orchestration.

Here are the list of attributes for the ItineraryService node :

• id : It has be a new GUID. Generate a new GUID and assign it.
• name : Any name that you want to assign to your Orchestration Service. This is the same name that would be used by the 'Receive Shape' in your Orchestration to subscribe to the messages comming for the Itinerary Service.
• type : This is the combination of Orchestration Name and Assembly Name. This information can be found by right clicking on the Orchestration in the BizTalk Admin Console, and click on Properties. It will show a list of properties. Under 'General' tab, you could find the Name and Assembly information. Just copy those two values seperated by a comma.
• scope : Since this is an Orchestration service, it's scope should be 'Orchestration'.
• state = "None".

A typical entry in the ESB.Config file will look like this :

"itineraryService id="5" name="TwoWayTransformation" type="ESB.MultipleWebServices.Orchestrations.TwoWayRouting, XactConnectArtifacts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=f922625124e336c0" scope="Orchestration" stage="None"/"

Once you have this information in the ESB.Config File, it should show up in the Itinerary Service Property Window for the ServiceName property. If it doesn't show, close the Visual Studio and open it up again... It should be able to show up now...

If it still doesn't, like it did for me, then the problem lies else where...

When you configure SSO using EsbConfigurationTool.exe, are you storing it in a datastore or a file???Change it to look it up from a file.... It will show up!!!! If you have configured it to look in a datastore then any changes made to the configuration file will obviously not show up...

So, Open up the ESBConfigurationTool.exe, click on Configuration node and see what is checked over there.... File Source or SSO Configuration Source???If it is SSO Configuration Source, then click on the UnConfigure Source button in the menu, Select File Source and then click Apply Configuration....

From now on, any changes made to the Configuration File will show up!!!!

That should set the ball rolling, as far as using a custom Orchestration in an Itinerary is concerned!!!

--

Preetham Reddy Chamakura

http://www.preetham-reddy.com/

ESB Toolkit : Tracking an Itinerary in the BAM Database

With version 2.0, BizTalk ESB Toolkit ships with a cool feature that lets you track the trajectory of various Itinerary Services, as they traverse through the BizTalk Engine. Ofcourse, It doesn't happen automatically. You've got to make it happen...

Every Itinerary Service has a special property called 'Tracking Enabled' which you've got to set it to 'True' during the design phase. This sets the stage for tracking an Itinerary Service. Note that, you've got to set this property for every single service in an Itinerary. Only those services whose 'Tracking Enabled' property is set to true will show up in the database. If you miss or forget for any of the services, then they won't show up....

All said and done.. So where does it show up??? When you install BAM Components during BizTalk Installation, they create a series of tables in the default SQL Server. One of them is the BAM Primary Import Database... Pull the records from the table bam_ItineraryServiceActivity_Completed and you should see all the Itinerary Services that are exectued successfully...

This table has lot of columns, but the one's that we are interested in are ItineraryBeginTime, ItineraryEndTime, ItineraryisRequestResponse, ItineraryState, ESBServiceName.

ItinearyBeginTime and ItineraryEndTime will track the point at which Itinerary is started and ended. ItineraryEndTime is null for all the services except the last one. That is the point at which Itinerary signs off. ItineraryisRequestResponse indicates whether it's a one way or a two way Itinerary. ItinearyState indicates whether it is 'Pending' or 'Completed'. It has the value of 'Pending' for all the Itinerary Services except the last one, which has a value of 'Completed'. This indicates that our Itinerary is successful and it's performing the way it's suppossed to be.

If you don't take proper care, when creating custom Itinerary Services using 'Orchestration' or 'Messaging', some of them might just return the values back to the On-Ramp without advancing the Itinerary. In those cases, the next Itinerary Steps wouldn't be executed, as a result of which, the Itinerary State remains 'Pending'. This is a good way to trouble shoot your itinerary services!!!

--

Preetham Reddy Chamakura

www.preetham-reddy.com

ESB Toolkit : Removing Encryption Certificates in the Itinerary Designer

Microsoft has shipped a new Itinerary Designer with ESB Toolkit 2.0 to assist the developers in creating the Itineraries visually... Creating Itineraries in ESB Guidance 1.0 was a real pain.... You would have to create that XML manually or using a .NET library and there was very little validation support.. Itinerary Designer in Version 2.0 comes with lot of cool features that not only simplifies the tasks but also has features for security and validation...

Every time you export the data either to an Itinerary store or to a file location on your local drive, the designer validates the XML and makes sure it conforms to the Itinerary Schema. It also has a way to protect the sensitive information in the Itineraries using X.509 Ceritificates.

While these security features are a welcome addition, sometimes it gets annoying during the development time. Most programmers, myself included, would want to concentrate on the task on hand rather than concentrate on trivial security features that doesn't really matter at the development time. So, If you are like me, and would like to disable Encryption while developing your Itineraries, then you've got plenty of options!!!

• In the Registry Editor, navigate to the subkey HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BizTalk ESB Toolkit\2.0\Designer, and then set the RequireX509Certificate property value to false.
• If you installed the BizTalk ESB Toolkit on an operating system that has 64-bit support, the subkey is HKEY_LOCAL_MACHINE\SOFTWARE\SysWOW64\Microsoft\BizTalk ESB Toolkit\2.0\Designer.

While, this is a recommended way to disable encryption certificates, I am not a great fan of interfering with System Registries. The way I've disabled in my system is by commenting out the lines in the Itinerary Configuration File which insist on throwing errors and warning for having an Invalid Certificate. Open up the file at C:\Program Files\Microsoft BizTalk ESB Toolkit 2.0\Tools\Itinerary Designer\ruleset.config and look for a property node that has a value of 'EncryptionCertificate' for it's 'name' attribute and comment out those following lines.

If you'd just like to disable the error, just disable for first 'Validator' node. If you'd like to get rid of warning too, then go ahead and comment out the next 'Validator' node...

That should keep the designer's mouth shut!!!!

--

Preetham Reddy Chamakura

www.preetham-reddy.com