Building your own in-house Business Intelligence solution 17 OCT 2022

Enterprise Business Intelligence done the right way versus developing a website to provide reports to your users

It is time to be honest about how Enterprises conduct their creation of technological solutions for their customers. Their customers maybe external or internal, many projects can go down paths that are are inert and are often harmful for the organisation. This happens for many reasons which this article will explain. If you want to avoid these pitfalls, definitely engage Info Rhino as a Business Intelligence solution provider to improve your organisation. solutions@inforhino.co.uk

Building a reporting website for your internal customers and external customers

• Why you are potentially going to make the biggest mistake ever and how to avoid it
• Enterprises have lots of data that their internal and external customers need

Traditional Business Intelligence and its evolution

Traditionally, enterprises would have an enterprise data warehouse, where data would flow into this EDW, to be made available for internet and and intranet users. the first attempts at making data available to users is via reports based upon spreadsheets and application based reports such as crystal reports, Jasper Reports, and other enterprise reporting solutions. how many years business objects was a compromise between reports that are developed by a team, and in other situations providing users with access to drag-and-drop, self-service business intelligence (SSBI). SSBI allows the Development Team to focus on the data being made available to a business intelligence solution that can allow users to define their own reports. The challenge here is that all but the most technical of power users still struggle to understand the data and how to manipulate it. We started to see more push towards dashboard based reporting such as Tableau, PowerBI, and SAP Hana. Many technical developers recognise the benefits of working on underlying data sources, and allowing power users to create dashboards for business users. This is very much about allowing users access to different levels of information within the organisation.

A key hurdle with Enterprise Business intelligence is the cost per seat, and the licence. enterprise Software technology providers have put hundreds of thousands of man hours into building these Business Intelligence Solutions, and they need to get paid for it.

Another challenge is that organisational data is constantly being updated, in many situations - overnight and periodic snapshots of data moving into a data warehouse, are not the best outcome for the organisation. it is worth highlighting that this is rarely as big a challenge as many business users will protest, but it can be a genuine problem for organisations wanting to have up to the moment information to hand.

A solution to these challenges brings the enterprise to decide to build their own web application capable of delivering reports to internal and external customers. The end result is a hybrid of an OLTP and OLAP solution. In doing this the very problem that business intelligence, in our opinion, sought to resolve is dispensed with. Business Intelligence seeks to bring data into a homogenous environment whereby it is structured for better decision-making analysis. Once the live transactional system is used for reporting without a degree of separation and correct structuring of data for reporting so begins the never-ending challenge of keeping the reporting system relevant and optimal.

To understand the issue with developing your own web applications to deliver reports to, users we must first of all understand examples of why this is done, how it is done, and then pick apart why it does not work.

We need to go down to real detail even down to the motivations of the necessary participants in what will become a failed experiment.

The stakeholders

C-level directors looking at their bottom line will see red flags when having to decide on which business intelligence reporting software to use. They can be faced with software that may cost millions of pounds a year in licencing costs. Additionally, the time to market in terms of new reporting requirements, that takes time for business users to define with business analysts, becoming business requirements for developers to implement, then testers having to confirm this has been developed acceptably, before then releasing to production. A more salient model is to have a small number of developers working with the business directly, in an agile fashion that will continually develop new requirements for the business. It appears to be an attractive proposition. Business users will like the idea of having their individual needs met by developers. This is the beginning of the problem.

The new iPhone conundrum

iPhone zealots will queue overnight to be able to get the latest iPhone. Does iPhone users get any decision on what the iPhone does? Absolutely no way. Now it may well be that metrics on device usage will feedback to put up designers and shape how the latest iPhone will be developed, but under no circumstances does iPhone users get any input into the design process. Why do organisations have significant amounts of resources spent on deciding user requirements that would never be practical for large organisations to develop for their customers? This has been our experience in most of our Enterprise on-site projects. Users having too much authority and say in how the product should be developed. One such example which seems minor, for one client, where they used a plug-in that had a drop-down list. They wanted the ability to have a cross on this drop-down list in certain circumstances, this software is out of the box, whilst customisable was not capable of delivering this. A long time was spent looking into this single feature that from a usability perspective did not make sense.

Co-dependent projects - Business and Development Teams

Enterprises are not aware that they are in a team that interacts with another team in a co-dependent manner. They both need each other for survival and end up with quite destructive attitudes towards each other. A good example is the user that asked for a feature that doesn't actually benefit the Enterprise, and the development team then implement it because this is what has been asked for. The development team will often realise this is not a necessary feature, but to keep the business user happy, will implement it anyway. the user will then test this feature which gives the Development Team things to do. We can see how these individuals are not doing anything of benefit for the organisation.

The mindset of most developers

Most developers like to write code. In many organisations this is how developers are valued - by lines of code written. This is a major policy because in our opinion most organisations should be removing code and reducing the amount of code that they write. They should always be balancing between introducing out-of-the-box software and deciding to write code for themselves. Another thing that most developers try to do is to make sure their skills are up to date, they are looking at the market in terms of their attractiveness to other employees, and will like to use more modern technologies so that they can potentially find new positions. This is a natural phenomenon not just in software development. Developers have a proclivity to doing liking new things, writing code, and producing new stuff. Developers often prefer building user interfaces themselves, as opposed to configuring reporting software and dashboards. Most developers join new projects where they get to create new software, using existing technology rather than building new technology is less interesting to them. The challenge is that developing software is a lengthy process, fraught with risk from too many moving parts and changing requirements.

The mindset of the customer

There should be no surprise that customers also like new things; new technology, new gadgets. New software promises to do things faster than before. It is worth highlighting that they can be regulatory requirements for software to change there can be new business opportunities to investigate, it is not as straightforward as saying that customers just want new things.

Project management teams

We know that most organisations have moved towards agile scrum Kanban project methodologies. We are not the biggest proponents of these methodologies and think that a more team oriented approach, where individuals just do what they need to be done, works better for an organisation and is scalable. Some form of enterprise vision and direction is needed, some monitoring of efficacy is required, but when these processes become too laborious, and detracts from the implementation of the software product.

The importance of control

At Info Rhino, we know that there are components and plugins which offers superior flexibility compared to older business intelligence reporting solutions. we know that we could find and a software house to develop specific components or do things ourselves and so there is an illusion of control. Those developing solutions from scratch will not really have much as much flexibility over the product that gets delivered.

The lack of transparency of Business Intelligence Solutions providers

We went online to look at some of the typical approaches of business intelligence solution providers.

• JasperSoft - no pricing link, we have to talk to sales.

• Dundas - Pricing link which involves talking to sales.

• Microstrategy - Pricing link, talk to sales.

Imagine going to a car showroom and not known what you going to pay for the car? Or going to a shop and having the cashier side what you're paying for the food? A software company that has built a product which is proprietary and cannot give a price for both per seat and concurrent licencing, is deceptive. When you are developing bespoke software this is an entirely different matter. Our experience of consulting to different industries shows that if you go to a vendor asking for pricing, an investment bank will get a different rate to a charity.

The human requirement and evolving requirements to enhance Decision Support

A significant question to ask is, do we need a human to interact with this information as a decision support process? It is an overlooked element of end-user reporting but on so many occasions businesses, to build systems around individuals and users when indeed this should be bought an automated process. At what point does a user need to make a decision?

An example of trading assets

Many people are now day traders and swing Traders. They rely upon looking at charts of price activity across different instruments, they use different indicators to try to identify trade signals, they may also with you map for information on different forms of Media. One of the challenges in being a trader is that you are human you need to sleep, eat, and drink. It can be argued that most Traders using technical analysis are really looking at mathematical information points. We can also say that this analysis of this type of information could be streamlined and automated. A system that is streamlined an automated will be a better human in many situations.

In the above example, we see what is truly required. We do not want to create a set of dashboards or systems that a user continually monitors on a daily basis. Instead, what we are looking for is a system that allows users to interact with data, their decision points can be framed and moved into an automated system that can then streamline future processing of information. Business Intelligence is no longer a process of creating reports for users to run, but a mechanism of automating knowledge into streamlining decision making.

Sticking with our example of trading, you may see that we are releasing a new platform called Crypto Statto. Our vision is to give users the ability to look at different market metrics, to also consume this information programmatically, thereby giving users the option to manually trade with better decision-making support or to automate this and execute trading on exchanges.

We seem to have arrived at a loose framework for what Business Intelligence should actually do. It seems that most reports are transitionary to eventually end up in systems that alternate processors helping to streamline the organisation.

Too often, a project starts out with the idea that there is data in a number of different systems that this data needs to be in a single data warehouse, and the end vision is for users to be able to access these reports. The missing step is that these reports themselves will eventually feed into automated streamlining of processes for the organisation. This is real the evolution of an organisation, continually improving and their users being an agent of change, rather than change for the benefit of their users.

Thinking about the disaster of self-built applications with data reporting built into them

We have established that the missing step in the Business Intelligence process is streamlining the organisation to the automation of reports into the decision-making process. There is a much more damaging element of organisations trying to build reporting solutions into the organisation - web application and reporting database.

Defining exactly when this happened, why this happened is a hard one. It seems to be that perhaps 12 years ago, organisations started to develop their own web applications to avoid needing to use BI Solutions. Our thoughts are that as Website technologies improved, developing your own solutions appeared to be slightly easier. Data modelling and Business Intelligence are essential facets of good information architecture. What seems to have happened is that application developers have been given the responsibility of creating reporting capabilities within enterprise systems. These developers don't understand databases to the level required, have little interest in developing these databases, and don't understand how to structure the information correctly, they are unable to use the advanced features of these data warehouses to optimise performance. This may not sound that bad. Do we really care if developing a bit more time building systems? Does it matter that they may not be as performant?

Unfortunately the inability of most of these teams to build effective reporting data models eventually kills the system.

Domain Driven Design in a an enterprise reporting solution

Without delving too much into the merits and demerits of DDD, one of the things we commonly see across different Enterprise is this commitment to continually updating software projects as new field requirements are issued. This results in development teams continually maintaining changing business needs, and the project architecture becoming highly complicated with duplication of code and methods throughout. There must be a process that is semi-automated to help manage data items without changing code in 5 or 6 places every time a change is needed. These solutions quickly become the "Big ball of mud".

Application Developers are rarely good data modellers and data developers

We repeatedly see database models for holding data that are completely unfit for purpose. We may see feature adverse code that fails to utilise newer features offered by database management systems. We see highly complicated stored procedures with hard coding that would never be allowed in application code. We see incorrect index definition, and homemade code that actually exists inside the database code natively.

Data Projects are best as generic and agnostic

When we undertake software projects, we tend to break them down into smaller chunks of work. We then analyse this, to try and estimate how long we think the project will take, finally attaching costings. The customer then has an idea on how much the customer project will take. This seems reasonable, but it can be catastrophic to implementing the solution that the customer needs. One obvious reason is the modularisation of the code implementation. By breaking items into small modules, we get a feel of progression - we see implementation over a period of time. We can report back to our customer and stakeholders giving them a feel that the solution is gaining traction. Sadly, what we are doing is dumbing down the implementation, we are making the solution a copy and paste affair, simply replicating fairly basic patterns which in the end makes modifying the solution highly complicated. A great example may be that we end up with 20 modules and then decide we need each of these 20 modules to have something extra within them. Assuming that all of these modules have been appropriately tested, we now introduce regression because we have to go through and check all the 20 modules again to ensure they don't break the solution. We start to add technical debt to the solution. Developers with more advanced capabilities, find themselves implementing low intelligent solutions that never reaches the boundaries of being innovative.

We are not building a house. We are not building a road. We are not simply getting materials, applying labour projections to that development, and estimating within fairly concise time frames and at eventual cost. What we are trying to do, is to build a system that is as flexible and as responsive to changing requirements as possible. This is not possible with typical Domain Driven Development, or as we tend to call it - Jenga Driven Development.

Our experience is that the best solutions are flexible and dynamic in nature, they are extendable, extensible and whilst we can never cover all scenarios, we allow the solution to add domain level requirements in a way that is not detrimental to the product.

One of the biggest threats to innovating within enterprises is the the budgetary and project-based approach. Project management teams are under immense pressure to make themselves accountable to the business and is most easily done by providing regular updates on modular implementations. The best implementations are those that provide flexibility going forwards. We cannot fail to ignore that most business intelligence reporting solutions are by their nature generic in an implementation. We see that enterprises must resist the temptation to implement modular solutions, and try to add dynamic generic approaches where possible.

The current state of Business Intelligence within enterprises when delivering solutions to their customers

We seem to be in an era of allowing fairly run-of-the-mill website implementations, with poor architectural design, data models of questionable standards, data warehouses that have all manner of performance issues. Behind the scenes, these systems are feature phobic, almost certainly of inferior performance, which will not meet the needs for the business.

We can also recognise that many of these Enterprise technology initiatives end up in a mudslinging over the wall affair, where each team within the implementation have quite an aggressive opposed to the other teams. At the centre of this is a lack of vision. The organisation should be thinking in terms of adding of turning manual processes into automated Processes, to then discovering other processes that needs to be streamlined to become automated. this is an iterative approach to to strengthening the business. The business should not be thinking in terms of removing those individuals because those individuals have useful insights into how the business operates, they can help shape new initiatives to strengthen and further advance the organisation.

What are our final thoughts on the use of proprietary software for business intelligence over using developing in-house reporting websites?

We see that many intelligent solution providers are obtuse in providing clear information on licensing costs, and are very expensive. Organisations does have to develop certain amounts off reporting capability within their information architecture. These solutions must be generic and dynamic in nature as much as possible potentially outsourcing certain development where it is known what the components should do. Enterprises should simply contact multiple BI providers, and ask them straight away for their costings by seat and concurrent licences.

Organisations should strive to avoid silos of strong business requirements that seems to require already existing functionality. there will always be duplication and you answers which doesn't exist how the organisation prize but looking outwards first before developing inwards is a powerful way to reduce costs.

Info Rhino often finds itself developing software for itself but we always look to do things through a generic approach, to then make that configurable which in turn makes the application less onerous to maintain. Companies should consider engaging companies such as ourselves. www.inforhino.co.uk/contact

Written with StackEdit.