High energy professional with comprehensive management experience in the software development industry for the Finance domain. Strong background in executive account management, mission-critical delivery management, human resources with an outstanding history of driving change from the first concept, through development, to implementation. Extensive international experience, including portfolios in North America, Europe and MENA regions.
Western Union completes 29 transactions per second on average. The received data is being integrated for statistical modeling and predictive analytics into a unified platform.
Like Western Union, thousands of other financial institutions and banks gather large data stores to use those in a variety of ways: to detect fraud, improve transaction processing and customer experience, boost trade execution, and gain a competitive advantage.
Today, banks come up to data processing with a special responsibility in the face of growing cases of sponsoring terrorism and money laundering.
No one has doubts that data is the new gold. The gold that should be stored and used respectively to bring its true value to your company. Being driven by new demands of local regulators, central banks, and ministries of finance, banks are those who are most in need of proper data storage.
Although a lot of banks are shifting from traditional data warehousing to Hadoop- or Spark-powered platforms for data analytics, there are still those who analyze information in monthly batches, which signifies 30-day trends can be lost.
All data should be managed fast and reliably, in a centralized place. Banks do succeed in reducing costs and skill gaps by acquiring cloud data processing services. However, neither Big Data technology nor cloud computing is no longer a competitive differentiator. Here is where banks are losing sight of what’s important. Banks should focus on accelerating data processing and turning the findings into actions to enhance customer service and customer experience, rethink risk measurement means, and handle fraud.
You can access and use data better by going to the cloud or implementing predictive modeling but that’s not enough. It would help if you comprehend the real power of data and data warehousing; thus, rethink the way you process, store, analyze it. You should create the storage from the ground up that fully matches with your business logic, your customer buyer journeys, transactions, payments, trade, marketing, and many more. Below are 10 tips to help you build that storage in the shortest time possible.
1. Understand the project goal and consider the time
The duration of a corporate data warehouse development project can take up from 2 to 5 years on average. Is it too long, isn’t it? Have you ever thought that when your DWH is developed, many things will be irrelevant? What if we would say that it’s possible to complete a DWH-related project in just 9 months? We have seen in practice that this is quite possible!
To shorten the duration of the project development, consider the project goal and values first. Articulate these things precisely and clearly so that everyone understands what outcomes you expect. The rest but not the least factors affecting the project duration include the appropriately chosen technology, the project roadmap, and the right sequence of jobs to be done, the allocated skills and resources. Critical data like Golden Source data and dictionaries should also be taken into account.
2. Analyze your existing information systems
Before proceeding with the development, you should take an inventory of all data assets. Driven by a “data first” mindset, you can reach project goals faster and stress-free. The success of the project depends heavily on data quality. To make decisions and handle reporting, managers need to have complete and reliable data and knowledge bases. Thus, it is required to analyze existing information systems, take inventory of data sources, define the global dictionaries, and the rules of building information links across various systems.
3. Make a rational choice of the technology platform
The analysis of existing systems, as well as a forecast of growth for the next 3-5 years, are important factors affecting the project duration. However, you should not overlook choosing the right technology platform and approach to data warehouse development.
Depending on your goals, you will have to choose between single-tier, two-tier, and three-tier architectures, the latter being the most widely used. Let’s say you need to delete redundant data and minimize the stored information, single-tier architecture will be your fit. In case you need to divide the data warehouse and available data assets, select the two-tier architecture. Ultimately, a three-tier architecture for data warehousing is the most popular as it has a multi-modular structure allowing for better flexibility and data management. Note that insufficient elaboration of the architecture or its redundancy can also negatively shift the timing and significantly increase the cost of the project.
4. Establish effective interaction between business departments
Each data warehousing module development and implementation requires involvement from a particular business department or mutual engagement of several departments. Ineffective interaction between the departments, misunderstanding the importance of data warehousing, high workload often become major things distracting from providing timely data or acceptance testing feedback. This is clearly an issue for all stakeholders to take up, as ineffective collaboration leads to launch delays.
5. Initiate data standardization procedures and data processing flow
If you are about to create a data warehouse from the ground up, there’s no doubt that something wrong is going with your data. You might definitely think of a centralized place to manage your data, if you face a variety of troubles: incorrect data format, poor description, missing data, different data formats in various departments, and many more. Let’s see how it works in practice: your front office system doesn’t capture the last name of the lead, whereas your digital marketing system requires the last name to be fed into a mandatory field. The gap between various data formats leads to poor data quality and the impossibility of making reliable data-driven decisions.
6. Audit all existing information systems to identify the real source of data
Most legacy banking systems were purchased from vendors. Let’s see, a bank would buy systems like an IBM zSeries mainframe, often referred to as monolithic systems, that had only one purpose, e.g., which was to support a deposit system. Buying one system per one purpose was a standard practice earlier for banks, which led to vertical silos and meant that each of those systems was written in different software generations. At the same time, it led to the interoperable complex web, which meant that systems were built and deployed first with no integration in mind. Consequently, it’s too hard to identify the origination of data sources.
7. Seek the support and collaboration of the executive management
Any enterprise-level project is doomed to fail without appropriate support and involvement from the executives’ side. Besides ensuring effective cross-department collaboration, executives are able to provide business sponsorship in data warehouse projects by communicating the value of data for the business area he or she owns. Business C-level representatives should also provide their requirements, e.g., what data they would like to feed to the analytical systems, what sources will data come from, what decisions it will support, and many more.
8. Use Agile methodology for data warehouse development projects
Waterfall is not a good fit for a data warehouse project. Consider Agile or Scrum to carry out the project jobs iteratively. For business intelligence projects, it’s impossible to gather and formalize all the requirements at once. You are losing time if a business analyst spends weeks to ask users what they expect to get from the system to be developed. They don’t know what they want. Seriously. We recommend to move progressively and get feedback on what’s been released in sprints. This approach allows for getting timely feedback and managing change requests more effectively.
9. Be prepared to embrace changes
Changes to data warehousing are inevitable. Eventually, you will have to make changes initiated by internal or external influencers, e.g., regulations or a new data management policy. Your ETL processes and toolsets should be established with possible changes in mind. Do comprehensive research to identify maximum possible scenarios and adapt your system to those beforehand.
10. Consider powerful memory devices for the highest performance
According to Oracle, input/output operations are often a bottleneck for large queries. Oracle paper says that modern hardware platforms can reduce latency and eliminate I/o operations. Such memory devices as DRAM and Flash can significantly improve performance.
Understanding the DWH project challenges and following the tips mentioned above, you can shorten the project development life cycle from a couple of years to 9 months. Surely, the timeline will also depend on how accurately you adhere to the project strategy, which is crucial for any IT project in general, and a data warehouse project in particular. Further, the implementation of recommendations requires close collaboration with a software vendor with strong data warehousing expertise for banks.
Guaranteed software project success with a free 30-minute strategy session!
How Diceus Can Help You
Diceus team will be glad to help you build the most effective architecture to manage your historical and real-time data and offers the following:
- Development and maintenance of a DWH project strategy
- Assistance in identifying the roles and responsibilities of a DWH environment
- Collection of business requirements
- Data modeling
- Assistance in selecting infrastructure tools
- DWH custom development and ETL program creation
- Integration testing and data loading
- DWH support
The complete list of DWH-related services is not limited to the points mentioned above and can vary concerning the complexity and specifics of your project.