Image of the glider from the Game of Life by John Conway
Skip to content

Managing LARGE Databases Continued

Well, I don't have much to report so far, other than still has a lot of bugs when it comes to it's Base program.  First off, before throwing the 3.4 million record Tennessee database at it, I thought I would try a much more compact database in dBase format with only a few thousand records.  Needless to say, it c...r...a...w...l...s.  If Paradox is like a cheetah running at 45 miles / hour, then SQL Server 200 is like a man lightly jogging at 11 miles / hour, while Base is a three-toed sloth taking a holiday. It is painfully slow.  And, when up and running, it constantly crashes, which means more time spent re-importing the database and starting over.  It blows!  Of course, the culprit is the Java Virtual Machine (JVM).

While the programmers at Sun Microsystems want to keep an application platform independant, deciding to base the entire infrastructure on the JVM is a poor decision.  While is runs at almost native speed when compared to Brand M Office, it has its limitations.  Base is one LARGE limitation.  Sure, I can develop robust, light and scalable database applications complete with reports, forms and queries.  Heck, it can even keep up with commercially driven database applications if you know what you are doing.  But begin to increase the size of the database anywhere past 1,000 records, and you have nothing but headache on your hands.  Which is unfortunate, as I love this application suite.

So, I won't even bother trying to import the Tennessee database using Base.  Instead, I am on to MySQL and PostgreSQL, as they are much more robust SQL applications.  Honestly though, I never expected to use Base to begin with.  I was just curious about it's speed, and boy did I find out.

{ 4 } Comments

  1. Ben | January 26, 2006 at 9:22 am | Permalink

    Base is supposed to compete against MS Access; it sounds like it has similar limitations as far as scalability and thus it comes as no big surprise that it doesn't measure up against Paradox, SQL Server, MySQL, etc.

    You should try SqlLite - it is small and VERY fast. Not as robust as MySql, but it sounds like it may be better for what you are trying to do. Give it a try.

  2. jordy | January 31, 2006 at 10:07 am | Permalink

    I think the goal to compete with Access is a noble one. I'll probably still use Postgress myself, but there is major demand for an open source Access killer. The open sourceness of it will make it not suck eventually, it just needs a little time.

  3. Dan | July 11, 2006 at 5:26 am | Permalink

    How did you get your data in to the SQL server?

    Using a DTS package direct copy is in my experience the fastest way to batch load your database and it can process 100's of thoughsands of records rapidly.

    Of course this does depend on your table structure e.g. A triggers, indexes , etc will slow it down.

    Have you tried Oracle (the free version... unless you're gonna splash out)?

  4. Aaron | July 12, 2006 at 10:40 am | Permalink


    I just used the import feature. I don't know the exact steps, but importing a dBase 5 table with 256 fields and literally millions of rows takes easily an hour. I was told by a Microsoft developer that the reason for this is SQL Server is mapping many more field types to every record than dBase 5 supports. For example, in dBase, it is either a number (int) or float. But in SQL Server, it could be a short, int, long, double or float.

    Eventually, I abandoned SQL Server for MySQL. It can import a dBase 5 table of that size in 10-15 minutes, which is much much faster than SQL Server 2000. And, being the FOSS advocate that I am, it follows my beliefs and principles, where proprietary software such as SQL Server 2000, does not. MySQL also does everything I need, plus much much more. I love it!

Post a Comment

Your email is never published nor shared.