Tag Archives: Best Practices
Rob Karel has been doing a nice job explaining Big Data, Metadata and other topics for Mom, so now I’d like to tackle another key group of stakeholders – your children. My kids have been asking me for years what I do at work. It hasn’t been easy to come up with an explanation that they can understand, so I usually just end up with something like “I go to meetings and stuff.” That works for a while, but it’s not very informative or inspiring. So if their friends ask “what does your dad do for work”, I can’t imagine what stories they make up. So here goes my attempt to explain to a sixth-grader what the job of a systems integration professional is. (more…)
I’m excited to share that, since its launch in January 2013, the GovernYourData.com community has been very well received. With over 4,700 unique visitors and nearing 600 registered members, many data management practitioners recognize it as a valuable go-to resource to support their data governance efforts. While maintaining our core objective of vendor- and product-neutrality, the site offers over 100 best practice blog posts from over 17 different contributors, shares the details on a dozen upcoming industry events, and has links to a wide variety of white papers, analyst research, recommended books, and other educational resources. (more…)
A front office as defined by Wikipedia is “a business term that refers to a company’s departments that come in contact with clients, including the marketing, sales, and service departments” while a back office is “tasks dedicated to running the company….without being seen by customers.” Wikipedia goes on to say that “Back office functions can be outsourced to consultants and contractors, including ones in other countries.” Data Management was once a back office activity but in recent years it has moved to the front office. What changed? (more…)
For some of you “old timers” in the IT industry, you will remember the days when we used to hand-code our own Database Management Systems. Of course today we just go out and buy a general purpose DBMS like MySQL, Oracle, dBASE, or IBM DB2 to name a few. Or, if we wind the clock back further, there was a time when we used to write our own operating systems. Today it comes with the hardware or we can buy an OS like UNIX, iOS, Linux, OS X, Windows, and IBM z/OS. And I can still remember hand-coding network protocols in the days before TCP/IP became ubiquitous. Today we select from UDP, HTTP, POP3, FTP, IMAP, RMI, SOAP and others. (more…)
Last week I described how Informatica Identity Resolution (IIR) can be used to match data from different lists or databases even when the data includes typos, translation mistakes, transcription errors, invalid abbreviations, and other errors. IIR has a wide range of use cases. Here are a few. (more…)
Whether you are establishing a new outsourced delivery model for your integration services or getting ready for the next round of contract negotiations with your existing supplier, you need a way to hold the supplier accountable – especially when it is an exclusive arrangement. Here are four key metrics that should be included in the multi-year agreement. (more…)
If you have been following publications in the Potential at Work Community or any number of Linkedin discussions such this one on the DrJJ group (a think-tank for information management best practices), you will have noticed the Agile methodology topic come up time and time again. For instance, check out the article Architect Your Way From Sluggish to Speed or the video Focus on Agility Adaptability. It hasn’t always been this way. For many years the architectural focus was on RASP.
In previous posts, we introduced the concept of the Informatica ILM Nearline and discussed how Informatica ILM Nearline could help your business. To recapitulate: the major advantage of Informatica ILM Nearline is its superior data access performance, which enables a more aggressive approach to migrating huge volumes of data out of the online repository to an accessible, highly compressed archive (on inexpensive 2nd and 3rd tier storage infrastructure).
Today, I will be considering the question of when an enterprise should consider implementing Informatica ILM Nearline. Broadly speaking, such implementations fall into two categories: they either offer a “cure” for an existing data management problem or represent a proactive implementation of data best practices within the organization.
Cure or Prevention?
The “cure” type of implementation is typically associated with a data warehouse or business application “rescue” project. This is undertaken when the production system grows to a point where database size causes major performance problems and affects the ability to meet Service Level Agreements (SLAs) and manage business processes in a timely manner. In these kinds of situation, it is mainly the operations division of the organization that is affected, and who demand an immediate fix that can take the form of an Informatica ILM Nearline implementation. The question here is: How quickly can the “cure” implementation stabilize performance and ensure satisfaction of SLAs?
On the other hand, the best practice approach, much like current practices related to healthy living, focuses on prevention rather than on curing. In this respect, best practices dictate that the Informatica ILM Nearline implementation should start as soon as some of the data in the production system becomes “infrequently accessed”, or “cold”. In data warehouses and data marts where the current month or two is being analyzed most often, this means data older than 90 days. For transactional systems the archiving cutoff may be a year or two, depending on typical length of your business processes. The main idea is to keep the size production databases from inflating for no good business reason and ‘nearlining’ the data as soon as possible without interrupting business operations or hurting the value of your data. Ultimately this should work to protect the enterprise from an operational crisis arising from deteriorating performance and unmet SLAs.
In order to better judge the impact of using either of these two approaches, it is important to understand the various steps involved in the “Nearlining” process. What do we find when we “dissect” the process of leveraging the Informatica ILM Nearline?
Dissecting the “Informatica ILM Nearline” Process
Informatica Informatica ILM Nearline involves multiple processes, whose performance characteristics can significantly influence the speed at which data is migrated out of the online database. The various processes are managed by the overall integrated nearline solution of Informatica coupled with a SAP Business Warehouse system:
- The first step is to lock the data that is targeted by the archiving process, in order to ensure that the data is not modified while the process is going on. SAP Business Warehouse does it automatically and you execute Data Archive Processes (DAP) for the cold data.
- Next comes the extraction of the data to be migrated. This is usually achieved via an SQL statement based on business rules for data migration. Often, the extraction can be performed using multiple extraction/consumer processes working in parallel.
- The next step is to secure the newly extracted data, so that it is recoverable.
- Then, the integrity of the extracted data must be validated (normally by comparing it to its online counterpart).
- Next, delete the online data that has been moved to nearline.
- Then, reorganize the tablespace of the deleted data.
- Finally, rebuild/reorganize the index associated with the online table from which data has been nearlined.
The Database Housekeeping process is often the slowest part of a Data Nearlining process, and thus can dictate the pace and scheduling of the implementation. In a production environment, the database housekeeping process is frequently decoupled from ongoing operations and performed over a weekend. It may be surprising to learn that deleting data can be a more expensive process than inserting it, but just ask an enterprise DBA about what is involved in deleting 1 TB from an Enterprise Data Warehouse and see what answer you get: for many, the task of fitting such a process into standard Batch Windows would be a nightmare.
So, it is easy to see that starting earlier in implementing Informatica ILM Nearline as a best practice can help to massively reduce not only the cost of the implementation, but also the time required to perform it. Therefore, the main recommendation to take away from this discussion is: Don’t wait too long to consider embarking on your Informatica ILM Nearline strategy!
That’s it for today. In my next post, I will take up the topic of which data should be initially considered as a candidate for migration.
If you haven’t already, check out the Potential at Work for Information Leaders site. We’ve just posted three great new articles designed to help you be more successful:
- “Driving value without locking down your data” Securing your data doesn’t mean inhibiting its use – far from it. Did you know that effective data masking practices allow information leaders to optimize the value data delivers to the organization while ensuring its security? Some forward-thinking information leaders are doing this and getting great results.
- “How fresh is your data?” Simply delivering data is not good enough anymore. You must get it to the right people at the right time while it is still fresh enough to be useful. Find out how to do it right.
- “Turn an application data migration initiative into a data governance pilot” A data migration effort can accomplish so much more than simply transferring data. Think about using it as an opportunity to improve the quality of existing data and apply new, higher standards to the information powering your organization.
Don’t miss out on topics that are key to your success. Please join the Potential at Work for Information Leaders community today. Available in nine languages, this site will continue to feature fresh, new ideas to promote the value of information management from a variety of top technology leaders.Sign up now!
We are excited to announce the new Potential at Work Community for Application Leaders.
As an application leader, you have a very demanding job. You have to successfully manage issues such as:
- Driving the maximum business value from your company’s enterprise application investments
- Keeping all of your enterprise applications current and meeting user requirements
- Delivering on your service agreements and managing all of the “ilities.”
- Defining an enterprise application strategy that includes on-premise and cloud
- Delivering timely, authoritative and trustworthy data for your enterprise applications
This community is here to help you to do exactly that and to help you to excel in both your current job and your career ahead. Our goal is to provide tips, insights, best practices and information from experts to help you become more successful.
Our first edition is focused on the theme of managing an enterprise cloud application strategy. For those who are in the process of selecting cloud application vendors, I’ve included a very handy Vendor Selection Checklist that is used by Informatica’s Vice President of Applications.
Are we interested in your input to the community? Absolutely! If you have an idea or content to share with the community, please contact us and we will get you published.
Join the community and start unleashing your potential by clicking on this link:
Roger Nolan firstname.lastname@example.org
Julie Lockner email@example.com
Click here for more information on the Potential at Work communities.