Essential Duties:

• Analyzes, designs and implements accurate, secure, efficient, logical and physical databases and coordinates database development applying knowledge of database design standards and database management system.
• Identifies and resolves production and/or applications development problems related to the use of the
• Database management system and end-user application software or utilities.
• Reviews project request describing database user needs.
• Studies existing data handling systems to evaluate effectiveness and develops new systems to improve production or workflow as required.
• Specifies in detail logical and/or mathematical operations to be performed by various equipment units and/or comprehensive computer programs, and operations to be performed by personnel in system.
• Estimates time and cost required to accomplish project.
• Determines if project requires creating series of new programs or modifying existing programs that access data stored in databases.
• Attends specifications meeting with project team workers to determine scope and limitations of project.
• Reviews procedures in database management system manuals for making changes to database, such as defining, creating, revising, and controlling database.
• Revises company definition of data as defined in data dictionary (information about data, including name, description, source of data item, and key words for categorizing and searching for data item descriptions).
• Determines and enters changes to data dictionary descriptions, including type, structure, and intended use of data within system, or assigns data entry work to programmer.

Job Requirements:

• Strong inter-personal and communication skills; capable of mentoring staff, acting as a vendor liaison, making presentations/leading meetings, and working closely with management.
• Ability to solve problems quickly and completely.
• Ability to identify tasks which require automation and automate them.
• A solid understanding of computer technologies and the direction of new technologies.
• Ability to plan and organize effectively.
• A solid understanding of the reinsurance and how technology can be applied to improve the business process.
• Maintains confidentiality of company-sensitive data.
• Keeps the Director informed of activities and situations, which will impact the achievement of corporate, and department goals and objectives.
• Performs other related duties as required to achieve the goals and objectives of the company and department.

Technical/Experience Requirements:

• Must have strong development life-cycle experience
• Must have SQL development experience with large databases
• Must have experience with a high level programming language in the development and delivery of complex software projects.
• Must have the ability to communicate clearly, concisely to non technical audiences

Supervisory Responsibilities: This position may have Supervisory Responsibilities

Contact: Jen Perri

All three charts span the last 12 months. Teradata is left off of the second chart so I can have finer granularity when comparing Netezza, Vertica, ParAccel, and Greenplum. Likewise, Netezza is left off of the third chart so I can have more granularity on Vertica, ParAccel and Greenplum.

It is important to note that according to Google: “The numbers on the graph reflect how many searches have been done for a particular term, relative to the total number of searches done on Google over time. They don’t represent absolute search volume numbers, because the data is normalized and presented on a scale from 0-100. Each point on the graph is divided by the highest point, or 100. When we don’t have enough data, 0 is shown. The numbers next to the search terms above the graph are summaries, or totals.”

Including Teradata

Needless to say, I was quite surprised to see the amount of searches regarding Teradata. It is interesting to note that the dip in October 08 followed by the sharp decline in January 09 follows an October 13th wire on “Teradata Breaks the Affordability Barrier to Analyzing Petabyte-Sized Data Warehouses” What I did notice is that the top searches for Teradata were technical in nature. IE: “Teradata SQL”, “Teradata date”, “Teradata Syntax” with searches rising on the topic of “Teradata Spool”, “Teradata Rank”, “Teradata SAP”.

Given the fact that the majority of the searches are technical in nature, it becomes obvious as to why the search volume was so high over the other products. Looking into search terms that are more exploratory or comparative are probably far more valuable from a marketing point of view. If I find some time, I may do another post that would analyze more “inquisitive” terms likely submitted by potential customers.

Omitting Teradata

When omitting Teradata you get a better sense as to where Netezza is relative to the other contenders. It is clear that Netezza is holding on to their position and slightly widening the gap as of late. I do however believe that the upward trend is related to the launch of Twinfin. If I am right, I would have expected to see more volume of searches. What is interesting is that unlike Teradata, The top search terms for Netezza are “Netezza Database”, “Netezza Teradata”, “Netezza Corporation”, “Netezza data warehouse” This is excellent news for Netezza as there seems to be an obvious interest and comparison between Netezza and Teradata. I believe my hunch is right regarding one of my earlier posts “Netezza Paved the Way“.

Netezza is well positioned against Teradata. Netezza has a large opportunity here because unless Teradata is being purchase with a whole host of other applications and logical data models, Netezza should be winning the analytics play. Switching from Teradata to Netezza is a hard sell even when the Teradata platform is fully amortized. A large portion of Teradata’s revenue is related to professional services but the nature of the search terms lead me to believe that Teradata is expanding existing environments and may not be seeing new client acquisition (for analytics that is). This is purely a guess but can easily be discovered from their filings. Perhaps Netezza should work on a migration strategy to chip away at the existing Teradata stronghold but definately attempt to take them head on for new client aquisition that fall into the extreme analytics category.

Omitting Teradata and Netezza

The pattern of Greenplum and Vertica trend-lines are so close in nature that it appears they are going head to head on most deals. Vertica does edge out Greenplum by a decent margin from an inquisitive search metric. The April 09 spike related to Greenplum seems to be related to the article on April 7th “GoldenGate Software and Greenplum Join Forces to Enable High Performance, Real-Time Data Feeds to the Warehouse“

Paraccel has a few good announcements lately but it’s tough to be in 4th position. Paraccel’s recent climb seems to be related to the Price Choper deal. I also expect to see climb in search volume in the month of October related to the announcement “Scales-Out – Merkle Scales-Out its ParAccel Consumer Data Warehouse Platform to 20 Terabytes“. From a search perspective, it still trails behind Vertica and Greenplum quite a bit.

Vertica has a steady upward climb going on related to search and all searches seem to be inquisitive in nature. IE: “Vertica Systems”, “Vertica Database”, “Sybase Vertica” Their recent release “Pink OTC Markets Inc. Cuts Reporting Time, Hardware and Programming Costs with Data…” is bringing the climb upward still. It is possible that another few deals would put Vertica in a position to break away into the Teradata, Netezza, Vertica trifecta.

Summary

My analysis needs to stop here for now. It would be nice to have the time to do a deep dive study on the technologies from a search, attitudinal, perception and sentiment point of view. This quick analysis was pretty interesting but it is just the tip of the iceberg and focuses on a small segment of data. This year there will be about 24 terabytes of social network data and news feeds. Carving out the MPP Database section of the web would prove to be an interesting study.

So in general this quick study leads me to believe:

• Teradata’s Search volume is mostly technical in nature possibly indicating that new customers are hard to come by
• Netezza is being compared against Teradata and the searches for Netezza are inquisitive or comparative
• Vertica is being compared against Greenplum and Sybase and all searches seem to be inquisitive or comparative

I just came in contact with a pretty good technical recruiter. He seems genuinely passionate about his career and asked me if I knew any hands-on Netezza resources. I thought I would put this out there in case you have an interest.

Our client, a New York City Investment Bank is looking for a hands on technical expert to work on a critical high availability platform. Must have experience with the following: Netezza data warehouse (1 year), Informatica (2 Years), SQL, PL/SQL and Oracle. Platforms include Solaris, Linux and Windows. Must have worked on the full SDLC. Major pluses include Cognos 8.2 and Autosys. The location is mid town NYC and the salary is between $100,000-$120,000 based on experience. If you are interested, please shoot your resume to jbnewman@genesis10.com.

Jeffrey B. Newman
Senior Technical Recruiter
Genesis10 – Business and Technology Consultants
212.688.5522 ext. 204
www.genesis10.com
http://www.linkedin.com/in/newmanlinkedin

• A bootable USB Stick
• A full blown Linux OS with GUI
• Evaluation Version of the Vertica Analytics database Fully installed and configured
• Apache/Tomcat Web Server
• LogiXML Dashboard Reporting
• Sample Data and Dashboards

Introducing the V-Stick

Powered by Vertica and LogiXML

We are currently putting the final touches on a sample dash-board and then will begin testing the Stick on different configurations. We have setup a forum specifically to support the V-Stick, so feel free to post your questions there.

Join the forum discussion on this post - (1) Posts]]> http://www.bcsolution.com/2009/07/vstick-coming/feed/ 0 http://www.bcsolution.com/2009/06/vertica-thumb-drive/ http://www.bcsolution.com/2009/06/vertica-thumb-drive/#comments Fri, 12 Jun 2009 04:42:24 +0000 cwardell http://www.bcsolution.com/?p=385 Can you believe it? A bootable 16 Gb thumb-drive with Linux, GUI, Vertica AND room for data!?

Yes, I admit it. I am a geek. But how cool is this. Imagine developing, prototyping, and presenting a data warehouse without procuring a server, installing Linux, or commandeering your hard drive. Everything you need is on a small bootable USB thumb-drive.

Welcome VStick

There were a few challenges in creating the VStick, but the complications were mostly around finding the right Linux distro.

Initially, I loaded some sample data and just about all the queries performed with sub-second response times. I am now in the process of cramming as much data on the stick that I possibly can. I am excited to see what the performance and compression would be like on the larger data set.

This little midnight engineering project really highlights the technology of Vertica.

Stay tuned, over the next few days, I will be loading and querying a larger data set and will let you know how it goes. Outside of evaluating and developing on Vertica, I was wondering if there was a market for a data warehouse on a stick? Perhaps the embedded market like Kiosks. 

Let me know if you have any ideas?

Regards,

Charlie

While I am on the innovation “rant” it reminds me of a few discussions I had with some of the brass at Netezza. I have been fortunate to have been invited to the Netezza campus a few times and we had a few blue-sky sessions. (This was before they got their new digs and their IPO.) My first impressions were great. The office had an entrepreneurial spirit and to my surprise the R&D team seemed to be well seasoned veterans. After talking with them for a while, you just knew they have been around the block once or twice.

My name is Charles, but my friends call me Charlie. A running joke with those who know me is that I am a terrible wine snob. Or should I say I am terrible at wine snobbery.  I know all the moves and can put on a good show but couldn’t tell the difference between 2 buck chuck and Grey Poupon.

As a joke one evening, Netezza had brought a bottle of two-buck-chuck along to dinner and I just happened to  love it. Needless to say they left me with a wonderful collection for my wine basement as well as a new nick-name from my friends at Netezza. A great memory and a pretty funny one, but I wanted to share one of the thoughts that came out of that evening. 

Netezza is a massively parallel processing appliance. In the Netezza world, your data repository is spread out as thinly as possible across as many Spu/Blades that it can. Each Spu/Blade is responsible for a small part of the entire data repository.

Let’s keep it simple for now and forget aggregation steps, summations, group by, etc. For my illustration, lets just say I want to perform a straight full table scan query against 1 billion records. If I have even distribution of 1 billion records across 100 SPU/Blades. The MAX LOAD or the MAX # OF RECORDS that any one Spu/Blade would have to read and filter is 10 million records.

In this particular example, a 1 billion record query would responds like it was a 10 Million record query. And the MPP is linear. If I had 200 Spu/Blades the same 1 Billion Record query would respond like a 5 million record query. This is MPP in a nutshell. I know that I am cutting out a LOT of details here but those details are not important for this post.

The innovation

Each Spu/Blade is essentially a dedicated computer that has its own dedicated:

• • CPU
• • RAM
• • Cache
• • I/O Channel
• • Network Adapter Card
• • Disk
• • and FPGA.

Although Netezza is focused on crunching enormous amounts of data, there is a huge opportunity waiting to be capitalized on. Netezza has opened up the FPGA. (Field programable Gate Array) to developers. The FPGA is programmed by a subset of the C language and compiled and deployed as a user defined function on each and every SPU/Blade. When each record passes through the FPGA it can be zapped by a native compiled User Defined Function.

You may be saying so what!? User Defined Functions have been around for as long as I can remember. What’s the big deal? How about Distributed Computing? What if the Netezza performance server was used as a massively parallel compute engine. The universities/academia, health research (IE:Cancer, aids, human genome), or the ever important search for extra terrestrial and large prime numbers could all be candidates for an MPP appliance in a box.

The next two paragraphs are from WikiPedia

SETI@home (”SETI at home”) is a distributed computing (grid computing) project using Internet-connected computers, hosted by the Space Sciences Laboratory, at the University of California, Berkeley, in the United States. SETI is an acronym for the Search for Extra-Terrestrial Intelligence. SETI@home was released to the public on May 17, 1999.[1][2][3]

Statistics

With over 5.2 million participants worldwide, the project is the distributed computing project with the most participants to date. The original intent of SETI@home was to utilize 50,000-100,000 home computers.[citation needed] Since its launch on May 17, 1999, the project has logged over two million years of aggregate computing time. On September 26, 2001, SETI@home had performed a total of 1021 floating point operations. It is acknowledged by the Guinness World Records as the largest computation in history.[12] With over 334,155 active computers in the system (1.8 million total) in 210 countries, as of August 04, 2008, SETI@home has the ability to compute over 528 TeraFLOPS.[13] For comparison, Blue Gene (one of the world’s fastest supercomputers) peaks at just over 596 TFLOPS with sustained rate of 478 TFLOPS.

Certain industries, like Life Sciences, pharmaceutical, financial, can’t go distributing data across thousands of volunteer computers worldwide. Procuring large grids to deploy their computational task is a challenging and costly task. Setting up a 50,000 node grid in a data center is not something most people would want to take on. The HVAC requirements for that size grid is just enormous.

The choice of a PowerPC CPU on the Netezza SPU/Blade was no accident. It runs significantly cooler than Intel or AMD. The reduced heat at the SPU/Blade level allows for a MUCH higher density per rack, smaller foot print on the data center floor, and reduces the HVAC and electrical requirement substantially. Come on Netezza! Where are the “Going Green” campaigns?

Netezza already handles the hardware, distribution, connectivity, and compiler for the SPU’s. As a result,  this may become a very viable approach for computational heavy studies. The next time your screen saver downloads a job packet from seti@home, think of the potential opportunity that Netezza could provide in finding aliens, the Human Genome project, FightAIDS@HOME

I wonder if Netezza could remove the DBOS and Storage and go-to market with a streamlined HPC Grid. (High Performance Computing Environment). Let me know what you think and  what industry you see the potential in.

That’s my blue sky for tonight.

Charlie (AKA: 2buckchuck)

When it comes to software though, I feel out of place unless I have a wrench in my hand. The hands on experimentation allows me to be confident and non “theoretical” when I propose radical paradigm shifts to my customers. Without challenging the “traditional” and being somewhat of a Rebol, I don’t believe I would have successfully managed to get both Teradata and Netezza deployed in the pharmaceutical industry. This was a huge milestone for MPP and considered a pretty major accomplishment. (Those of you who know pharma know first hand of the Oracle domination.)

By the way, Rebol above is not misspelled. It’s another “Secret Agent Man” technology I keep in my arsenal of weapons..

So what’s my next tinker?  I have many on my list. But I am closing in on something pretty radical. Without fully disclosing what mad scientist hack I am up to, the least I can do is tell you that It has something to do with Vertica and a thumb drive. This will be helping in a future project of mine. (More details to come)

Stay tuned  it may not be what you expect? The details should be posted within the next week or so.

Best Regards,

Charlie

Today you see a lot of traction with columnar databases. Rightfully so, when you understand the columnar approach and the minimization of I/O offered by technologies like Vertica it is a no-brainer to give it a try. Are we seeing a paradigm shift today as we saw when Netezza first launched their MPP appliance? Probably, but I think the road for Netezza is still pretty golden.

Netezza trumps the MPP space. Going public has validate not only the company but the technology. They have taken the survivability and risk of adoption question out of the picture. In doing so, Netezza has cleared the field and paved the way for technologies like Vertica and Paraccel.

My guess is that over time, Netezza will win quite a few Teradata clients. As the capitalization of the Teradata investments come to an end, the need for increased storage capacity will grow. Faced with investing into more nodes, companies would be wise to re-evaluate their Teradata strategy and consider Netezza. (Given of course they are not locked into Teradata by purchasing the additional logical models or applications that run on it)

Netezza will grab market share from Teradata and in doing so they just may keep NeoView in the gate for years to come. Oracle just doesn’t understand this space so at present, I don’t see any competitive threat from Larry for MPP architectures. I wouldn’t be surprised if Oracle made a bid for one of the Columnar databases though. (I would prefer however; that they stick with OLTP instead of misleading people into believing they can build large scale Query and Reporting databases on OLTP technologies.)

If my hunch is correct, this opens up a tremendous opportunity for technologies like Vertica. I believe Netezza sees a great opportunity for C-Stores and surprisingly may welcome the competition to the space.

I believe Netezza is focusing on the “prize”. I would assume that they want to be known as “The Brand” for Petabyte data warehouses. Technically there may be no size limit with Vertica, Paraccel, or Sybase. It is however my feeling that if an organization is crunching that much data, they will lean toward Teradata or Netezza.

So I believe Vertica and Paraccel owes Netezza a few “props”. The success they are about to have is related to Netezza’s work in this space.

In my next post, I will talk about a niche that Netezza may have outside of the data warehousing arena.

Stay tuned..

If I had a transactional system like Facebook, MySpace, Travelocity, Expedia, it would make much more sense for me. The data requirements are large but the data is accumulated over time, transaction by single transaction. Ad hoc queries, reporting, and analytics that happen on VLDB’s, usually, or at least, should not be performed against operational transaction based systems. There is usually an extract of the transactional data and it is housed in either a more suitable schema or better yet or more suitable VLDB technology.

Here are a few questions and  challenges you may face:

• • • How do I get terabytes of data from my operational system  to the commodity clouds effective
  • • I am not sure I would want to say to my customers: “Just send your data to Amazon.com and I will take it from there.” Imagine how many hands it needs to touch before it gets to the floor of the data center.
  • • Are you in an industry (financial, pharmaceutical) where regulations are such that a multi-tenant grid would be frowned upon?

I definitely see the positive side if the data was already at the cloud, but typically, we receive data via ftp in many different formats at different frequencies. They all need to be standardized and purified before it gets into the Cloud DB. So from my point of view, data warehousing is not a real good candidate for the Cloud.

I love the idea of the cloud and would welcome differing opinions. Am I missing something?

Join the forum discussion on this post - (1) Posts

I have never been able to achieve performance on multi-terabyte databases using SMP (Symmetrical Multi Processing) servers and OLTP databases. In my opinion, the relational/OLTP databases like Oracle, SQLServer, MYSql, etc.. are just the wrong tool for adhoc queries, analytics and reporting of VLDB’s (very large databases). The results I obtained when utilizing MPP have been staggering when compared to the response times of query and analytics of OLTP databases.

There is no theoretical size limit on databases based on MPP architectures. (Massively parallel processing). The more nodes, SPUS, blades, or pizza boxes you have in your cluster, the more capacity your database has and the faster it will perform.  With that said, I am pretty confident that Teradata, Netezza and Vertica can scale to the Petabyte size. There seems to be no real inhibiting factor that I know of.

So what does one choose?

I maintained a Teradata environment for about 6 years and have gone under the hood pretty deeply with Netezza and Vertica. I have programmed at the FPGA level for Netezza and had the opportunity to pick the brains of the senior engineers at Vertica for a while. I am pretty confident that at this point, the MPP technologies I mention below should handle the workload.

The question comes down to “What is your budget and business need?”

Teradata

Teradata has a wide range of logical models and applications they can sell along with the WH. Teradata supports transactional processing and ad-hoc queries very nicely. Teradata also has a professional services team which contributes a large portion of their revenue, but all this comes at a premium price.

Netezza

Netezza is a beast of a machine that does full table scans extremely well, their FPGA technology is very impressive, and the zone indexing does a nice job in minimizing I/O. Netezza’s secret sauce is in their FPGA (Field programable gate arrays). Not to long ago, Netezza opened up the FPGA to developers and now give tremendous power by creating FPGA based UDF’s (User defined functions). These UDF’s perform exceptionally well and can put the afterburners on your query and analytics if done correctly. Netezza has proven the test of time and has a pretty impressive client list and is a very viable alternative to Teradata.

Vertica

Vertica is relatively a new comer to the VLDB space and they are definitely the one to watch. I like the technology and the fact that Vertica is led by Stonebraker who is a veteran in the industry. ALso quite notable is Jerry Held (Exec from Oracle and Tandem) who sits on the board.  You will need to grasp a slight paradigm shift in the way their database operates but it supports ansi standard SQL and ODBC connections so you shouldn’t have any problems. Vertica’s compression algorithms are extremely clever and when combined with a columnar data store they are breaking records. Vertica will definitely give you an A-HA! moment once you understand how the magic happens.

By the way, Syncsort and Vertica broke the world * TPC record for data loads at a fraction of a price when compared to Netezza and Teradata.

* A clarifying point on the TPC record. This test was not an instance of the TPC-H benchmark but rather a benchmark on load performance based on the TPC-H data produced by DBGEN.  The benchmark was audited by a 3rd party, published, and available for others to contest.

Summary

Technically they will all crunch the data and they will all do it MUCH faster than OLTP based databases. Some questions you will need to consider:

• • Q) Do I need or am I looking at other logical data models and applications to kick start my project?
  • Q) Do I want to leverage a professional services team?
  • Q) Do I need user defined functions or stored procedures?
  • Q) Does purchasing proprietary hardware scare you?
  • Q) Do I have a substantial budget or is this a warehouse on a shoestring?
  • Q) Is my DBA adaptable?
  • Q) Do I have a mixed workload?
  • Q) How many users will be hitting the database concurrently?

I know both Vertica and Netezza offers proof of concepts.

TAKE ADVANTAGE OF THE POCS!

One approach you may want to consider is to start with a technology like Vertica. They offer a free trial and it installs easily on commodity hardware. This will provide the lowest cost of ownership and it just may fit your needs.  In parallel, work on setting up the POC with Netezza. This may take a little longer because staging a Netezza environment within your walls has many logistics involved.  Choosing the right technology is a crucial step in the development of a Very Large Data Warehouse. You should take your time on this critical phase and choose wisely.

Join the forum discussion on this post - (1) Posts