It’s Time For You To Get Your Big Data Organized


If you have been reading about “Big Data”, obvious questions are do I have it, and if so, where do I put it, how do I find it, and how to I use it?

If the Volume, Velocity, Variety of your data has increased significantly over the past little while, you probably have Big Data.

Using this guideline, individuals at tablets, laptops and PCs can have Big Data, not just large Corporations. Corporations will typically have Big Data at servers and in the Cloud.

At a practical personal level, you probably have a few thousand .doc, .pdf, .xls, .ppt files, e-mail messages, 100 application system executables and fifty URL “favorites”.

Sure, you can (1) “find” e-mails in Outlook using add-on products such as Lookeen, (2) separately find files using Windows Explorer, (3) find application executables via desktop shortcuts and (4) find URLs under the “favorites” button on browsers you use.

Difficult, time consuming.

The problem, of course, is you are likely to have a cluster of .doc /.xls and URLs that relate to a particular Project. How do you find these? No guarantee, of course, that a particular file will only be needed at one cluster.

I have 10 Tb of files across four drives, many of these files are video recordings of stage events plus content destined to feature in one or more documentaries. I found by moving to a Kbase that I could easily accommodate and manage 10,000 objects from a single computer screen, and, with the exception of images, videos and sound recordings, if I am able to think of a couple of words likely to be part of the content, I am able to find what I am looking for. A repeat of the same exercise outside of the Kbase can easily take an hour, often longer.

Who needs Windows Desktop?

Key words that you invent and encode to documents provide little benefit – the mindset you had at the time you encoded a document is not likely to match your current mindset. Not much choice but to use key words for video/audio recordings so key words do serve a purpose.

Here below is a screenshot of a Kbase that puts a primary focus on the U.S. Dept of State “Country Reports”, but includes other information as well.

Let’s do a search to find the addresses of foreign embassies in Washington.

We will start with “Massachusetts Avenue”.

Some of the content is local to the Kbase, some at URLs.

US01

Notice that “hits” are highlighted.  Notice the difference between SQL searches and free-form searches. The latter tells you what it found plus what it did not find.

us02

If we click on “Australia” we see that this node has both an attached document and a URL link.  You could park 100 objects at this node.

You can browse the “hits” via a Find button.

US03

us04

Practical Considerations

What about your individual needs in respect of Big Data?

The search capability seems impressive but much of the data in the State Dept Kbase is static.

Your needs are likely to be more dynamic so you may derive greater benefit from organizing all of your “documents” given that the number, content and required focus can change daily.

If your desktop looks like this, it’s time for you to take a serious look at Kbases.

You can quickly acquire the ability to find needles in haystacks.

Kbase03

About kwkeirstead@civerex.com

Management consultant and process control engineer (MSc EE) with a focus on bridging the gap between operations and strategy in critical infrastructure protection, healthcare, connect-the-dots law enforcement investigations, job shop manufacturing and b2b organizations. (C) 2010-2017 Karl Walter Keirstead, P. Eng. All rights reserved. The opinions expressed here are those of the author, and are not connected with Jay-Kell Technologies Inc, Civerex Systems Inc. (Canada), Civerex Systems Inc. (USA) or CvX Productions.
This entry was posted in Database Technology, Decision Making, Enterprise Content Management, Strategic Planning and tagged . Bookmark the permalink.

One Response to It’s Time For You To Get Your Big Data Organized

  1. Limits? We don’t know.

    There are no apparent practical x,y size limits to “sheets” – the problem you get to if you define too much real estate is you would need to zoom out to the point where you could only see the shapes of clusters and you would need to have a grid overlay in order to directly navigate to any cluster.

    For sheet sizes that measure less than say 4×4 ft there is a hand navigator that lets you scroll the entire sheet as a single bitmap. The sheet scrolls pretty much as fast as you can move the mouse.

    As for numbers of records in the underlying database, we have built sheets (Satellite demo http://www.civerex.com/pages/cmanlar.html) comprising 7,000 objects within a single database that included several other sheets with similar numbers of objects.

    Accordingly, chances are search times would not start to slow down until you reach 50,000+ objects. We will be adding context indexing that should extend the number of quick access objects per sheet to over 500,000.

    Few individuals or corporate functional units are likely to want to consolidate 500,000 objects to one screen..

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s