Eight Phases of Planning My Project - Assignment Example

Project Progress In my project proposal I had planned to do the project in eight phases as listed below: Gathering Information for the paper System design and analysis Detailed analysis of factors involved in reading databases Details of building a data mining system and running classification algorithms against test data Decision tree algorithms and implementation Data mining system sequence analysis Implementing the project life cycle Submit the final paper after final review Each of these phases was to be done in a week and thus the entire project should have taken eight weeks to do and complete. The following is the status on these phases. Gathering information This was arguably the most time consuming part of the project, data mining being a new field there is a lot of information with over lapping views and ideologies, some schools of thought were also conflicting. I wound up sifting thorough many books at the library and there was a host of information on the internet in regards to this topic. The main challenge was to find sources and references that tallied and complemented each other on most aspects. This is the phase in which I decided to settle on WEKA as the data mining tool of choice for the project. Reasons for this are discussed in the project report. Also I decided to bring up to speed my JAVA programming skills since according to my the information I had gathered I needed to write code in JAVA. System design and analysis In system design and analysis I had to do flow charts and look at varying designs and architecture of the similar systems from the more than twenty sources I collected and cited. After reading about Occam’s razor I decided the simplest implementation was the best and settled for a data flow diagram built inside WEKA’s knowledge flow environment. WEKA’s knowledge flow environment was in and of itself complete and eliminated the need or use of long and tedious flow charts to guide most of the data flow for the coding process. Detailed Analysis of factors involved in reading databases Getting data is the first and most important step of the data mining process and for this reason I had to look around for credible and useable data sources. Since most databases on the internet are in MYSQL I read up on MY ODBC database drivers that allow JAVA code to interact with a MYSQL database and make downloaded the drivers ready to use for this part of coding when it came to coding. Details of building a data mining system and testing decision tree algorithms I found out more about decision trees, decision tree algorithms and their application in real life. I also read about the C4.5 algorithm by ROSS QUINLAN since it was the family of algorithms I was going to be using in this implementation. Thanks to many WEKA tutorials on the internet, it was not too difficult to find the details for the steps necessary to build a complete data mining system. WEKA also came with some sample data and I ran tests for the decision tree algorithms. These tests and their results are contained in the project report. With these results and tests I could now do a more comprehensive data flow diagram and so I redesigned the data flow diagram. Decision tree algorithm and implementation I settled on WEKA’s in built class called J48 which implemented C 4.8 a more advanced algorithm as compared to C 4.5 that I had decided to use at the beginning, since WEKA came with this class inbuilt there was no need to ‘re-invent the wheel’ building my own algorithm in JAVA. I settled on this class and implemented it in the knowledge flow environment. System sequence analysis Armed with knowledge from the previous weeks of research and testing it was not too difficult to settle on what model to go with for System sequence analysis. WEKA’s in built data flow environment also helped ease this problem because it gave a good visual representation of the program flow. Implementing the project life cycle This part entailed some feasibility study as to which database I could get data from automatically with some code I could write. Since accessing databases online needs the database administrator’s approval I had difficulty getting any database administrator to give me either a password or a username with limited permissions to only read their data bases for obvious security concerns and no service I found offered any API (Application programming interface) calls to access weather data over the internet. I how ever came across the Utah State University weather centre with data archives on the internet and downloaded this data set, formatted it manually and used it in my implementation. Finally when writing code I stumbled upon JYTHON a JAVA implementation for PYTHON and decided that PYTHON was easier to learn and use and so instead of going with JAVA opted for JYTHON for extending WEKA’s capabilities. While working on the project I found many future capabilities for this project and noted down some future improvements. That is maintenance in the software development life cycle. Submission of final report The final report has already been submitted to the lecturer concerned and a round of evaluations done. I made the relevant corrections and this stage is still on going. Highlights and challenges Like any other project there were high lights and challenges with this project. The challenges and highlights are discussed below. Lack of well defined standards and supervising bodies in data mining As I discussed in the project report data mining has become a buzz word and catch phrase and many tasks that do not warrant the title data mining are being called data mining. Consequently there is floods and torrents of materials on the topic. I had difficulty choosing what should be considered data mining and what was not. I even wrote about it in the project report under what is data mining and what is not data mining. There were not too many standards and I could not immediately establish who the authority in data mining was. With standards like industry standard for data mining (Crisp-DM) offering different definitions of terms and standards of what data mining was. I however looked through my sources for a common thread of thought and settled on the one I discuss in my project report. Programming In my original project plan I had decided to extend some of WEKA’s capabilities and call classes from JAVA. Again even in this field there were many tutorials most of which were vague and nondescript. Many code samples from the internet either did not work or were too shallow and with the nature of JAVA this phase of my project was tedious and time consuming. Database access As I discussed above it was difficult to find any API calls available for data information and the closes I got to getting real time or fully automated data extraction from databases online was from a website called infochimps.com that promised data archives with just a registration and I thought maybe I could build a robot (web-bot) to automatically retrieve data online and make my system more useful. Unfortunately the information form infochimps.com was in mostly inconsistent data formats and even though it was promised to be free I could not download a single data set. I stumbled upon the data set form Utah State University that I use in the report. Unfortunately this data set had to be downloaded manually and post prepared manually to fit the mold of the data mining operation. I discuss this in the project report. Utilizing open source software While there are numerous advantages of utilizing free to download open source software there are also challenges to go with it. This project was implemented on an open source tool for the advantageous reasons fiscal implications being one of them. The problem I had with using a free to download open source tool is that there was no dedicated support for the program and when I ran into a problem I had to depend on online peer forums to fix the problem. The other reason is because the software is downloaded I did not know many of the internal ‘mechanisms’ of the program and had to do the tedious process of reading line after line of JAVA code to try and decipher what happens on the front end of the software. WEKA was easy to use A pleasant surprise was the ease with which I could use WEKA and get for free state of the art data mining algorithms with ease. WEKA came with both a command line interface (CLI) and a graphical user interface (GUI) making the development process much easier. There was also many useful tutorials and PDF manuals from The University of Waikato website to help ease the process of developing the system for data mining. WEKA was truly and amazing tool and the completion of this project may not have been timely if WEKA was not employed. JYTHON bindings Another pleasant surprise was the fact that WEKA allowed me to write my code in PYTHON thanks to JYTHON technology. This cut development time by more than half and put at my disposal many free to download and use open source resources since I could interface using PYTHON. Native PYTHON classes Earlier I stated that when coding using JAVA I had to find and download the correct MY ODBC drivers to interface with JAVA and then find the correct API calls to make from JAVA so as to access a database. PYTHON comes with native classes to access MYSQL databases and can be comfortably used to access different versions and flavours of MYSQL. These native PYTHON classes are said to be faster on python.org. Future Possibilities For this project I chose to do this project because of my interest in the field of data mining. After working on the entire project I have discovered many possibilities for this project implementation. Extend the possibilities One possibility for this project is to expand it and instead of being just a system for data mining databases on weather move it extend to more everyday uses like predicting traffic on some roads, crime patterns, the stock market prices or future prices of commodities like oil. If this is done using python it can be made available as a thin client over the internet via a web browser or alternatively through API calls to a mobile application running on Google’s Android Operating System. It could also be used as an application on social media for example to find trends in who is most likely to be friends, who is most likely to buy a certain product when it is advertised to them or who when they endorse a certain product could make it go viral. The usefulness of such knowledge is apparent and if data mining works as well as this project shows, the commercial possibilities are endless. Businesses will gladly pay to get information like who is most likely to buy a certain product with a higher accuracy percentage. This knowledge will help further speed up a knowledge based economy. It can make advertising not just a broadcast exercise hoping some one will be interested in your product but rather fronting your product to a potential buyer. Such information can be mined from data we currently hold. Just some of the useful uses we can put data mining into. WEKA implementation in PYTHON Thanks to my new found interest in PYTHON I am considering studying data mining algorithms deeper and then writing a replica of WEKA’s functions as a PYTHON library with relevant API calls, whether this should be a commercial or open source implementation is still n the workings. This library can be called form any other python implementation whether it is a game built with PYGAME or the more immediate use of interfacing with GOOGLE APIs. Google supports PYTHON as the programming language of choice and provides many APIs that can help a developer extend Google’s functionality. Sourcing for a data donor Since data mining needs large amounts of data for it to be more accurate, I am still sourcing and sending out emails and proposals owners of large databases on weather. This will help validate this system further. The idea is to find a data donor willing to give large amounts of data that can easily be formatted to fit the data mining operation preferably automatically by a python script. This project I hope can be adopted by an open source society and developed into a more useful solution like predicting weather patterns so farmers dependent on rain fed agriculture especially from third world countries can be more productive and avoid huge losses that they are currently prone to due to changing weather patterns. WEKA’s little flaw Though WEKA was a great tool and the success of this project relied heavily on WEKA there was a little flaw I came across, I think this may be because WEKA was developed in a University so was built more for teaching than actual data crunching. When large amounts of data were fed to the system it was painfully slow and often hang or produced errors. Figure 1 WEKA out of memory error message The above message came up when I ran the data set I got from Utah State University and even after increasing the heap size it still did not work. This may be one of the reasons I would want to build my own data mining tool modeled on WEKA so that I always understand the internally workings of the tool and so can easily fix such bugs. Read More