In my last post, I asked the question “Is Web Data Possible?” At first, this question may seem obvious, but closer inspection of the challenges in making web data consumable makes it apparent how difficult the problem really is. It also highlights why it hasn’t happened yet, despite many attempts to do so.At Datafiniti, we’re making web data truly available for the first time, and we’re fascinated the possibilities opening up such a data source represents. What would you do if you could get instant access to all web data? It’s a question that touches on the possibility of accessing almost all human experience and knowledge instantly. It’s incredibly exciting, but also difficult, to think about its impact.
Where We Are Now
The concept of instant access to all web data is still in its infancy. Businesses are already realizing the fruits of data-driven processes and decision-making. Most of this has occurred by using information that’s already available from internal systems – CRMs, SCMs, ERPs, etc. But as more phases of the customer’s journey goes online, the more of that customer’s data is native to the web. This has rapidly resulted in the web becoming the largest repository of customer preferences, interactions, and comments, causing Doug Laney of Gartner to comment that the web is the largest database for any company.“Web scraping” is how most people refer to accessing web data, but this method is incredibly incomplete and error-prone. It doesn’t produce web data in any usable sense. It just produces a simple copy-and-paste log file. Without any refinement through sanitization, aggregation, and other data enrichment techniques, it provides a poor representation of the web data needed by most organizations. Yet despite the poor data quality it produces, it does provide some value and is a popular choice for acquiring web content.
The Next Phase
So, if current approaches of using a small sliver of web data are already providing some utility, what could the potential of instantly accessing ALL web data hold? Right now, it’s difficult to forecast its impact, but we know it will be huge.The most immediate effect is obvious: web data will significantly improve any business’ ability to react to market changes.Businesses that thrive are those that are nimble, efficient and responsive to the market. However, all of that is only possible if businesses can access comprehensive information on their customers’ motivations, competitors’ offerings, and overall market ecosystem. Unfortunately, this data, when available, is often incomplete and not current. One way to supplement this critical data set is to leverage web data. The large aggregation of consumer and competitor web data will provide insights that internal company data collection methods would be hard-pressed to deliver. Web data fills the large data gap that exists today for almost every business. Filling that gap means better insight into customers, competitors, and the market as a whole.Like I said, all of the above is the immediate effect of web data. What comes next has the potential to change how our society as a whole behaves.We’ve already seen how enabling instant access to single points of web content have revolutionized our society. Google has effectively made the web an extension of every person’s own knowledge. Now apply this same concept to businesses. What happens when the web is an extension of every business’ own database? There is a next-generation of applications and analytics waiting to be imagined and released once web data is a reality.
How You Can Learn More
I’ll be sharing some possible ideas and prototypes for this analysis during the upcoming NewCo Austin event. Register here to attend our presentation on May 29th, 2015 at 9:30 am at our downtown offices. We’d love to have you over!