You arrive in a foreign city for a couple of days and you need to get some stuff done and attend some meetings. You have no idea how to move around and you don’t speak the language, nor you are able to drive. Yet, people approach you to sell you a car, competing on the features and capacity of their offering, then also giving you the opportunity to take some lessons so as to be able to take ‘full advantage’ of it. You end up paying a lot of money for the high-end car and wasting your stay in attending driving seminars; yet you still are not able to navigate around and you missed those meetings, you are finally told that the city is to blame.

compass - a photo by flickr and user maldita la hora

The above story probably sounds peculiar and the approach taken is awkwardly wrong; after all it doesn’t take that much of intelligence to simply get a taxi or hire a driver for those two days. Still and to our surprise, this is how the enterprise world works when it comes to selecting a data mining solution or tackling a data intensive problem, and we’d like to serve our fair share to try and shed some light, given our very limited experience thus far. Here’s the original version of the story:

A big enterprise wants to leverage the big amount of data it collects so as to get actionable recommendations with regards to some pressing topics of its operations. The company does not have the talent, nor the experience and expertise to handle that, so it starts looking for external partners. Typically, these end up to be big software vendors and their certified partners, providing ‘holistic’ and ‘customizable’ solutions to every specific problem on earth. The managers get excited; these solutions seem to provide everything they need to feel comfortable about, including a complete stack of algorithms, full integration with their CRM and existing intra-enterprise infrastructure, also a well-known brand and tons of phone support staff. So, they proceed to sign the really fat check and assign the rest to their colleagues.

It takes a few weeks for the vendor to set everything up, then the various modules turn out to be quite complicated and the manuals non-readable at all, so the company proceeds to also pay the extra specialized training partners to educate some selected people of its personnel on how to import data and what to click for an algorithm to operate. Now that they know how to put the tools into action, the company is finally able to get back to the original problem and try to come up with a working solution. But, guess what, they own the Ferrari, they know how to drive it, but they have absolutely no clue on how to navigate across the chaotic city, or interpret the various signals to get the job done!

Data mining software is not the equivalent to, say, Microsoft Office. If you have some basic Word skills, you’re probably able to write and edit a text document; however, if you know where to find the Support Vector Machines button in a data mining suite, you’re still far away from coming up with any working solution to your problem. Trial and error, standing at the core of anything related to data mining, requires a deep understanding of the underlying processes behind each one of the numerous algorithms, enabling you to follow best practices on every step, pick the right instruments and be able to interpret the results so as to proceed to the right direction and improve, from naive results to really insightful ones. After all, in experimental problems, the tools are not the ones which matter the most; actually they do not matter at all if there are no experts to put them in good use.

At the end of the day, experts are needed to tackle such sophisticated problems, and our humble recommendation is to start with them, and then decide on which tools you need to acquire, if any at all. You may also consider that, in most cases, data mining is not a task that you need to pursue on a daily basis; capable resources are required into solving a problem once, then you just replicate its solutions across time. Moreover, since you have acquired good results, there’s typically no need for an expert to replicate these into new data sets; you may just run an already available black box of code, or implement some fully transparent recommendations using naive tools, like advanced filtering. After all, it typically makes much more sense to get a taxi to move you around than hire a full-time driver and have him waiting till the next time you happen to be in town.

We do believe that the above may help you to get a better understanding of how things work, or actually should work, in the industry and what dead end solutions you need to avoid. In any case, we’d happy to show you around and help you find your way next time you’ll get lost with data or you happen to land in the far-off area of data mining, rest assured that there lies really great value waiting to be discovered!

To us, new formats mean new opportunities. And we’re trying to experiment with different ways of spreading our message, starting with the very basic ones. Newsletters that is, and we came up with the original content of our website, in a format and shape that we bet you’ll like. Here it goes, feel free to spread it around and we’re awaiting of your positive feedback to make it happen more often, as a magazine occurring periodically, it’s your pick!

We submitted a press release at PRweb earlier today, the full text is also reproduced below while you also may download the release here (.pdf). 

It is with joy and happiness to welcome you in our brand new start-up. Its name? mineknowledge!

We are a group of young and passionate data engineers and we’re putting human intelligence and our high expertise in between of various advanced machine learning algorithms, to finally provide you with the very secrets trapped in your data sets, unveiled.

You may find out more about us, the problems that we address, the solution we provide, the hassle free process to get this solution working for you, plus a descriptive example of all these. And we are going to feed this blog with new case studies, demonstrating how we conduct business and deliver on our promise, while we are already thrilled of you being here and we are awaiting for your precious feedback!