Recent Developments in Data Warehouses and Its Application in E-Commerce - Article Example

Conferences in Research and Practice in Information Technology Recent Developments in Data Warehouses and Its Application in E-Commerce John F. Roddick School of Computer Science, Engineering and Mathematics Flinders University of South Australia PO Box 2100, Adelaide 5001, South Australia roddick@csem.flinders.edu.au Abstract The data warehouse provides a platform for effectual data extraction/mining. In essence, the data warehouse comprises: exhaustive and abridged information, integrated data, metadata, and historical data. Arguably, all of these components improve the data extraction procedure and the views for achievement. Data marts, such as online analytical processing (OLAP) are an architectural annex of data warehouse for the reason that the mart have tailored data and runs on data at an extreme level of summarization. Data warehousing is a tactical enterprise and IT scheme in various organisations at the moment. In this regard, the review paper seeks to provide an insight on the recent developments of data warehouse and its technical application in the field of e-commerce.: .Keywords: Data Warehouse, Business Intelligence, OLAP, E-commerce 1 Introduction A data warehouse is the notion of data mined from functional systems and made accessible as historical snapshots for impromptu queries and listed reporting. In essence, features that differentiates data in the data warehouse from that established in the functional setting is that it is: structured in a way that pertinent data is grouped mutually for unproblematic access (Su et al., 2009). Furthermore, numerous copies of the data from different points over time are reserved jointly, and once the data is stored in the data warehouse it is not restructured. Instead, the historical snapshots stockpiled in the Data Warehouse are occasionally invigorated with data from the functional databases. According to Vela et al. (2012), the key issues addressed by a data warehouse are that clients endure hard time generating ad-hoc or other focused reports and queries and. This is brought about by numerous aspects such as the majority of the data is stockpiled in Adaptable Data Base System (ADABAS), which is prickly for clients to access. In addition, the data stores were intended for business procedures and not ad-hoc reporting. Another aspect is that achieving the data often needs waiting for a programmer to either offer a tailored downloaded program or build up the report and at other times every data may not be reliable as of the same point in due course (Hwang et al., 2004). According to Wu et al. (2001), data warehousing is a cluster of decision support tools, intended to enhance the knowledge worker, such as director, manager, and market analyst to make enhanced and quicker decisions. Essentially, the previous five years have observed a volatile progress in Data warehousing, both in terms of products and services provided, and in the acceptance of these technologies by manufacturing industry. Data warehousing technologies have been set out productively in numerous industries, such as producing industries for order consignment and client support, retail for inventory administration and customer profiling, monetary services for analysis in risk, credit card, claims and scam detection. Other industries that have adopted Data Warehousing include utilities for power application evaluation, transportation for fleet management, healthcare for results analysis, telecommunications for both fraud detection and call analysis. Schneider (2008) study presents a guideline of data warehousing technologies, concentrating on the unique prerequisites that data warehouses direct on database management systems (DBMSs). Schneider (2008) further notes that a data warehouse is a subject-oriented, incorporated, time changeable, non-impulsive set of data that is employed mainly in managerial decision making. Characteristically, the data warehouse is kept independently from the business’s functional databases. In essence, there are numerous reasons for doing this, for instance, the data warehouse brace on-line analytical processing (OLAP), the operation and functionality specifications are rather distinct from those of the on-line transaction processing (OLTP) applications conventionally braced by the functional databases. Technically, OLTP applications often computerize office data processing assignments like banking transactions and data entry that are the fundamental daily processes of a business. Tan et al. (2003) assert that these assignments are prearranged and cyclic, and comprises of petite, minuscule, segregated transactions. In addition, the transactions need comprehensive, modern data, and interpret or modernize a few records accessed often on their main keys. Constancy and retrieve-ability of the database are vital, and making the most of transaction output is the fundamental performance metric. As a result, the database is developed to reflect the functional semantics of recognized applications, and, especially, to reduce concurrency inconsistencies (Hwang et al., 2004). Fig 1: Data Warehouse Components (Su et al., 2010) 2 On-line analytical processing (OLAP) OLAP denotes the manner wherein business customers can access information through intricate paraphernalia that permit for the course-plotting of dimensions like hierarchies time or. Arguably, Stefanovic (2013) posit that OLAP offers multidimensional, abridged insights of business information and is employed for reporting, examination, scheduling and modelling for business optimization. Essentially, OLAP methods and paraphernalia can be employed to work with data warehouses developed for complicated BI systems. These systems according to Glorio et al. (2012) process queries needed to determine trends and examine vital aspects. Reporting software produces collective sights of information to continue informing the management concerning the condition of their enterprise. Other BI paraphernalia are employed to amass and examine information, like information extraction and data warehouses; and document management and document warehousing (Glorio et al., 2012). 3 Quality and Data Warehouses Schneider (2008) defines data quality as part of functionality over expectation, or as the loss conveyed to people from the instance the product is dispatched, in short data quality is the "suitability for utilisation". Essentially, the nature of this description openly means that the theory of data quality is relative, for instance, data semantics, which is the elucidation of data, is distinct for each distinct user. Schneider (2008) mentions that the predicament of data quality is essentially entwined in how customers in reality employ the data in the system, given that customers are the eventual judges of the quality of the data generated and if no one in reality employs the information, no one will ever pay attention to enhance its quality. Fundamentally, as a decision support system, Su et al. (2010) posits that a data warehouse must offer sophisticated quality in data and services. Viscidity, novelty, precision, ease of access, ease of use and functionality are in the midst of the quality aspects needed by the data warehouse end user (Su et al., 2010). Yet, most stakeholders’ excessively are implicated in the data warehouse lifecycle; every one of them appears to have their quality specifications. As previously stated, managers often use an OLAP query technology to obtain answers appealing to him. What’s more, a decision-maker/manager is often worried with the quality of the stockpiled data, their appropriateness and the ease of querying them using the OLAP paraphernalia. Furthermore, the Data Warehouse manager require amenities such as fault reporting, metadata ease of access and comprehension of the data appropriateness, to facilitate detection of variation and reasons behind them, or issues in the stockpiled data (Tan et al., 2003). In addition, the Data Warehouse Designer should assess the quality of the schemata of the data warehouse setting both present and recently generated and the metadata quality too. Besides, the designer requires software assessment values to examine the software packages he wants to procure. Vela et al. (2012) posits that the Data Warehouse modules programmers can make excellent utilisation of software enforcement standards so as to achieve and assess their work. Metadata reporting can as well aid their work since they can evade errors connected to schema data. Based on this examination, Tan et al. (2003) securely argues that distinctive roles entail unlike compilation of quality features, which should be preferably taken care in a reliable and consequential manner. From Su et al. (2010) study, it is argued that data quality is of extremely one-sided nature and should preferably be taken care in a different way for all users. Simultaneously, the quality objectives of the concerned stakeholders are extremely varied in nature; thus, they can be neither evaluated nor attained openly, but need multifaceted measurement, calculation, and design methods, frequently in the shape of an interactive procedure (Wu et al., 2001). Conversely, the reasons for information paucities, non-accessibility or attainability issues are certainly objective, and rely often on the information system characterization and enforcement. In addition, the forecast of data quality for all users must be founded on objective quality aspects that are calculated and contrasted to users' prospect. The issue that crops up, then, is how to systematize the design, management and development of the data warehouse in a manner that all the distinct, and occasionally differing, quality specifications of users can be concurrently fulfilled (Stefanovic, 2013). As the amount of clients and the intricacy of data warehouse systems do not authorize to achieve total quality for all users, another issue is how to prioritize these specifications to facilitate customers’ satisfaction concerning their significance. Su et al. (2010) posit that this quandary is often exemplified by the objective design of the data warehouse where the quandary is to locate a set of materialized sights that optimize customer demands reaction time and the universal data warehouse repairs cost all together. 4 Business Intelligence The hasty tempo of modern’s business setting has made Business Intelligence (BI) systems essential to the achievement of an organisation. Technically, BI systems turn an organisation’s raw data into applicable data that can assist administration recognize vital trends, examine user behaviour, and make intelligent organisation decisions hastily (Pabreja & Datta, 2012). In the past few years, BI systems have for long been employed to recognize and tackle office necessities like effectiveness and output. Presently, business organisations are increasingly employing BI to examine client behaviour, recognize market inclinations, and look for novel opportunities. What’s more, BI depends on Data Warehousing, which is a data storehouse developed to brace organisation's management, making commercial storing and administration of warehouse information vital to any business intelligence and data warehousing (BIDW) systems solution (O'Leary, 2011). Devoid of an effectual data warehouse, business organisations cannot retrieve the information needed for data analysis punctually to enhance convenient decision-making. Pabreja and Datta (2012) posit that the aptitude to attain information in instantaneous has turn out to be progressively more vital in modern years for the reason that management cycle times have been narrowed considerably. Furthermore, competitive demands need organisations to make intelligent decisions founded on their received business information as well as doing it swiftly. Pabreja and Datta (2012) believes that the capacity to turn raw information into functional data in an apt way can present hundreds of thousands even up to millions of sterling pounds to firm’s end result. BI helps in tactical and functional decision making, for instance, a survey by Gartner ranked the strategic BI utilisation as business functionality administration, and client relations optimization. Others include, supervising organisation activities and conventional decision Support Packaged objective for definite functions or strategies and administration BI reporting (O'Leary, 2011). Business intelligence (BI) has two fundamental distinct meanings connected to the utilisation of the phrase intelligence. O'Leary (2011) posits that the main, less often, is the individual intelligence ability used in business processes. According to Su et al. (2010), BI is a novel domain of the analysis of the use of human cognitive abilities and simulated intelligence tools to the administration and decision reinforcement in various business issues. The subsequent associates to the intelligence as data treasured for its prevalence and significance. Vela et al. (2012) holds the view that BI is professional data, comprehension and paraphernalia resourceful in the administration of managerial and entity business. Hence, from this context, business intelligence is an extensive class of functions and tools for collecting, offering access to, and examining information with the intention of assisting enterprise users make enhanced organisational decisions. Vela et al. (2012) further notes that the term means having an all-inclusive comprehension of all of the aspects that influence the business. Furthermore, it is vital for organisations to have a comprehensive comprehension concerning aspects such as the users, competitors, fiscal setting, and internal functions to make efficient and excellent value business decisions. According to Pabreja and Datta (2012), a focused domain of business intelligence recognized as competitive intelligence concentrates exclusively on the outer competitive setting. Data is collected based on competitors behaviour and decisions are made in relation to this information and diminutive if any consideration is paid to collecting internal data In contemporary businesses, mounting values, computerization, and technologies have steered enormous quantities of data becoming accessible. In addition, Data warehouse technologies have developed storehouses to stock up this information also enhanced Extract, transform, load (ETL) and even lately business application incorporation paraphernalia have augmented the quick gathering of data. OLAP reporting tools have permitted more rapid production of novel reports, which evaluate the information (Hwang et al., 2004). In this regard, BI has now turned out to be the art of filtering through enormous quantities of data, retrieving relevant data, and whirling that data into knowledge whereupon measures can be taken. 4.1 Issues in BI The core key to triumphant BI system is merging information from the various distinct enterprise functional systems into an organisation data warehouse. Presently, extremely few firms have a full developed organisation data warehouse because of the enormous effort scope towards merging the whole organisation data. Pabreja and Datta (2012) stress that considering up-and-coming extremely dynamic business setting; just the mainly competitive organisations will accomplish continued market triumph. The firms will differentiate themselves by the capacity to influence information concerning their bazaar, clients, and functions to take advantage of on the business opportunities. Numerous surveys such as those conducted by International Data Centre, Forrester and Gartner report that the majority of the organisations across the world have the desire to invest in BI. It was noted that in spite of main investments in customer relationship management (CRM) and enterprise resource planning (ERP) throughout the last ten years most organisations are still under pressure to accomplish competitive advantage (Stefanovic, 2013). O'Leary (2011) believes that this was brought about by the information confined by these systems and any company would focus for one objective called ‘accurate access to data hastily’. Therefore, the organisations must brace the analysis and use of data so as to make functional decisions. Say for smudging cyclic products or offering specific suggestions to clients, companies require right access channel to data hastily and enforcing smarter business functions is where BI affects the end result and brings back value to any organisation. 5 Recent Development in Data Warehousing According to Gartner, Inc (2012) CIOs should make themselves acquainted with the fundamental trends in data warehousing and how they will influence the technology cost-benefit balance set out to distribute business analytics value (Stefanovic, 2013). In essence, Gartner analysts hold the view that data warehouse will remain a fundamental module of the IT infrastructure and interest in business intelligence (BI) and the broader class of business analytics heightens, optimization, supple designs and optional plans will turn out to be more imperative. Presently, data warehouse continues to be one of the prime information storehouse in the enterprise and Tan et al. (2003) believes that if CIOs are conscious of the fundamental market trends and how up-and-coming technology solutions will merge with established practices then they can evade budget waste through 'misdirection' by the data warehouse administration. Some of the fundamental data warehousing trends include: Optimization and Functionality: According to O'Leary (2011), sophisticated performance for hardware administration of input/output (I/O), CPU/memory balancing, and disk storage and are now built-in roughly as a course matter in data warehouse competent basis. O'Leary (2011) argues that some novel entrants are concentrating on optimization as a discriminator and almost all data warehouse retailers are at the moment tackling the problem of optimizing storage for the warehouse through application established data assignment strategies and data compression. In addition, retailers are also using up enormous effort distinguishing their products on functionality claims and technology, in means that are not inevitably noteworthy to the use case. Data Warehouse Appliances: While there are various reasons why firms reflect on procuring an application, the key reason is minimalism. According to Pabreja and Datta (2012), the retailer develops and verifies the configuration, balancing software, hardware and services for a knowable functionality. What’s more, the appliance is delivered complete and inaugurates in haste. Thus, if there are any issues, a call to the appliance retailer is the first guiding principle, but there is a resulting consequence too, in that appliances can accelerate delivery by evading protracted hardware balancing. The Rigorous Proof of Concept (POC): Arguably, during 2010, according to Gartner (2012) most organisations paid attention to Gartner's previous recommendation to carry out a proof of concept (POC) with "selected” vendors during the choice stage of the data warehouse database management system (DBMS). In essence, Gartner suggests that proof of concepts employ authentic source-system extracted data (SSED) from the functional systems to the probable extent, while executing the POC with various users, generating a workload of data warehouse that looms that of the setting to be employed in manufacture (Stefanovic, 2013). Data Warehouse Mixed Workloads: Schneider (2008) posits that there are six workloads that are conveyed by the data warehouse base: vital reporting, bulk/batch load, crucial online analytical processing (OLAP), instantaneous/incessant load, data extraction and functional business intelligence. In this regard, warehouses conveying all six workloads must be evaluated for certainty of mixed workload functionality as lack of schedule for mixed workloads will steer augmented management costs in due course, as capacity and supplementary workloads are included, potentially steering immense maintainability problems (Schneider, 2008). Column-Store DBMSs and the Resurgence of Data Marts: Column-store DBMSs usually display quicker query reaction than conventional, row-established systems and can operate as outstanding data mart basis, and still as a key data warehouse base. Vela et al. (2012) predicts numerous retailers shifting the software pricing paradigm from a more conventional per-core or per-user paradigm to a price founded on the capacity of information loaded into the database. On its part, a data mart is defined by Vela et al. (2012) as an application-explicit analytic storehouse of any size, usually with a definite, minor group of user’s contrary to data warehouse. Fundamentally, data marts can be employed for data warehouse optimization by divesting fraction of the workload to the data mart, recurring superior functionality to the warehousing setting. Data Warehouse as a Cloud and as a service: Gartner in 2011, observed that data warehouse as a service came in two tastes: outsourced data warehouses and software as a service (SaaS). According to Vela et al. (2012), in the cloud computing, data warehouse is first and foremost an infrastructure design preference, as an information paradigm must still be built up, a combination plan must be deployed and business intelligence user access must be permitted and administered. What’s more, private clouds are an up-and-coming infrastructure design selection for some firms in bracing their data warehouse and analytics. Today’s data warehouses are comparatively self-effacing in volume, storing about ten terabytes (10 TB) or a reduced amount. According to Stefanovic (2013), a less percentage of respondents, in his study (4%), confirmed that they have data warehouses with in excess of a petabyte of data. Yet, things are changing rapidly, in that 82 percent of respondents account that the number of data in their warehouses has augmented over the years. In the midst 13 percent believe this augment has been significant, with the data volume in their data warehouses heightening by more than 50 percent in the last one year. According to Vela et al. (2012), enormous firms are far more probable to possess hundreds of terabytes of data within their data warehouses. Statistically, 22 percent of companies with more than 5,000 workers account data warehouses of this range, against only 5 percent of smaller organisations. In a separate Independent Oracle Users Group (IOUG) survey performed lately established that further than data warehouses, in general, data is approaching or above petabyte levels athwart various business enterprises: this includes information both inside and outside the data warehouses (Pabreja & Datta, 2012). Additionally, almost 30 percent reported settings beyond 100 terabytes, and 10 percent had more than 1 petabyte (1,024TB) of disk-resident data. What’s more, numerous organisations are still merely gaining knowledge concerning the benefits of both business intelligence and data warehousing. Glorio et al. (2012) posit that organisations are at the moment still gaining knowledge on how to control the data warehouse and there have been several genuine noticeable wins for the companies achieved from the data warehouse, but still it is underutilised. Based on Glorio et al. (2012) study, it is noted that the business intelligence and data warehousing are comparatively novel notions to the company; thus, most organisations must to develop into an enterprise-broad data warehouse, with views into all aspects of information. Arguably, the effect of big data will more and more be endured in the data warehouse space and viewing one year to the fore, most companies expect data augmentation in their data warehouses in the scope of 1 percent to 25 percent (Vela et al., 2012). While the big Data dare turn out to be veracity, several organisations see issues ahead for their existing systems. Actually, just about 55 percent of the companies interviewed by Vela et al. (2012) were certainly sure that they are convinced their present data warehouse settings will manage to scale to correctly examine big Data in the future. According to Pabreja and Datta (2012), companies know that they sit on potentially enormous data resources, and they must comprehend in regard to novel approaches to successfully tap into these precious data stores. 6 Data Warehouse for E-commerce The adding of e-commerce to the data warehouse comes in with both intricacy and novelty to the plan. E-commerce is by now acting to amalgamate one time unconnected transaction processing systems like the sales system, the advertising system, the catalogue system, and the consignments system all must be available to each other so that the e-commerce to operate efficiently through the internet (Triantafillakis et al., 2005). Internet’s extensive usage has compelled various business organisations to venture in e-commerce and since it is a two way process: business-to-consumer (B2C) and business-to-business (B2B) then everybody can take part on the online business. E-commerce provides enormous business opportunities to both small scale and large-scale industries, as many companies now want to host their enterprises online to reach the novel market after they failed to reach efficiently with their advertising campaigns or sales team. Presently, many custom brick and mortar enterprises have generated an online presence, which permits their customers to shop virtually any time they want (Yu et al., 2009). Consequently, by committing their resources online, most organisation not only anticipates an increase in sales, but also a heightened public awareness of the organisation’s products, and optimistically more customers. Arguably, the internet is infiltrating every society and economy aspect, consumer have become aware of the possible risk in utilising this technology. In this regard, this awareness has negatively affected most online-based organisation, resulting to additional efforts on the organisation part to persuade custom customers not only to shop physically, but also to apply the internet for procuring and general information collection (Yu et al., 2009). Furthermore, online based shops provide individuals with the expediency of buying and selling products and services from the soothe of their residence and contrary to the shopping malls, online shops permit individuals to acquire products anywhere and anytime (Triantafillakis et al., 2005). Other exclusive concerns comprise capturing the course-plotting tendencies of its clients, tailoring the Web pages or Website design, and comparing the e-commerce part of its business in opposition to directory sales or authentic store sales. In this regard, data warehousing can be used for all of these e-commerce explicit discussed issues and provides an excellent platform on how to capture the data. The benefits brought about by e-commerce data analytics are enormous and apparent for all online vendors to observe (Triantafillakis et al., 2005). Data warehouse fore-commerce provides users with an established, business-steered solution that assist e-commerce based businesses to enhance easy comprehension of their customers, concentrate their advertising and promotional operations and magnetize repeat clients. According to Yu et al. (2009), there are a numerous means that e-commerce user’s benefit from their available data subsequent to working with data warehouse: attain sales context per product; capacity to examine their customers; and assist with the client connection administration. In addition, data warehouse provides more insight into shopping activities and consumer decision-making and enhance capacity to aim promotion campaigns to entity customers. 7 Conclusion Conclusively, it has been observed that data warehousing, which is a cluster of decision support tools, enhance the knowledge worker, such as director, manager, and market analyst to make enhanced and quicker decisions. Besides that, some of the notable trends in data warehousing includes optimization and functionality; rigorous proof of concept; data warehouse appliances; data warehouse mixed workloads; data warehouse as a service and cloud; and column-store DBMSS and the resurgence of data marts. What’s more, by designing a data warehouse purposely for an e-commerce business, an organisation can manage to control progressively imperative competitive edge, especially in knowledge management provided by that decision support systems. As it has been confirmed factual in innumerable data warehouse schemes, the e-commerce need to offer an explicit concentration to the data warehouse to be developed on, bearing in mind the fact that some exclusive issues comprise capturing the direction-finding routines of its customers. In conclusion, data warehouse comes with novel capacity to the organisation, and helps manager run the organisation efficiently, and increases the competitive advantage of the organisation. 8 References Read More