Comments on: Managing LARGE Databases Continued http://pthree.org/2006/01/26/managing-large-databases-continued/ Linux. GNU. Freedom. Sun, 07 Sep 2008 23:31:59 +0000 http://wordpress.org/?v=2.7-hemorrhage hourly 1 By: Aaron http://pthree.org/2006/01/26/managing-large-databases-continued/#comment-988 Aaron Wed, 12 Jul 2006 16:40:17 +0000 http://www.pthree.org/2006/01/26/managing-large-databases-continued/#comment-988 Dan- I just used the import feature. I don't know the exact steps, but importing a dBase 5 table with 256 fields and literally millions of rows takes easily an hour. I was told by a Microsoft developer that the reason for this is SQL Server is mapping many more field types to every record than dBase 5 supports. For example, in dBase, it is either a number (int) or float. But in SQL Server, it could be a short, int, long, double or float. Eventually, I abandoned SQL Server for MySQL. It can import a dBase 5 table of that size in 10-15 minutes, which is much much faster than SQL Server 2000. And, being the FOSS advocate that I am, it follows my beliefs and principles, where proprietary software such as SQL Server 2000, does not. MySQL also does everything I need, plus much much more. I love it! Dan-

I just used the import feature. I don’t know the exact steps, but importing a dBase 5 table with 256 fields and literally millions of rows takes easily an hour. I was told by a Microsoft developer that the reason for this is SQL Server is mapping many more field types to every record than dBase 5 supports. For example, in dBase, it is either a number (int) or float. But in SQL Server, it could be a short, int, long, double or float.

Eventually, I abandoned SQL Server for MySQL. It can import a dBase 5 table of that size in 10-15 minutes, which is much much faster than SQL Server 2000. And, being the FOSS advocate that I am, it follows my beliefs and principles, where proprietary software such as SQL Server 2000, does not. MySQL also does everything I need, plus much much more. I love it!

]]> By: Dan http://pthree.org/2006/01/26/managing-large-databases-continued/#comment-974 Dan Tue, 11 Jul 2006 11:26:59 +0000 http://www.pthree.org/2006/01/26/managing-large-databases-continued/#comment-974 How did you get your data in to the SQL server? Using a DTS package direct copy is in my experience the fastest way to batch load your database and it can process 100's of thoughsands of records rapidly. Of course this does depend on your table structure e.g. A triggers, indexes , etc will slow it down. Have you tried Oracle (the free version... unless you're gonna splash out)? How did you get your data in to the SQL server?

Using a DTS package direct copy is in my experience the fastest way to batch load your database and it can process 100’s of thoughsands of records rapidly.

Of course this does depend on your table structure e.g. A triggers, indexes , etc will slow it down.

Have you tried Oracle (the free version… unless you’re gonna splash out)?

]]> By: jordy http://pthree.org/2006/01/26/managing-large-databases-continued/#comment-49 jordy Tue, 31 Jan 2006 17:07:36 +0000 http://www.pthree.org/2006/01/26/managing-large-databases-continued/#comment-49 I think the goal to compete with Access is a noble one. I'll probably still use Postgress myself, but there is major demand for an open source Access killer. The open sourceness of it will make it not suck eventually, it just needs a little time. I think the goal to compete with Access is a noble one. I’ll probably still use Postgress myself, but there is major demand for an open source Access killer. The open sourceness of it will make it not suck eventually, it just needs a little time.

]]> By: Ben http://pthree.org/2006/01/26/managing-large-databases-continued/#comment-42 Ben Thu, 26 Jan 2006 16:22:03 +0000 http://www.pthree.org/2006/01/26/managing-large-databases-continued/#comment-42 Base is supposed to compete against MS Access; it sounds like it has similar limitations as far as scalability and thus it comes as no big surprise that it doesn't measure up against Paradox, SQL Server, MySQL, etc. You should try SqlLite - it is small and VERY fast. Not as robust as MySql, but it sounds like it may be better for what you are trying to do. Give it a try. Base is supposed to compete against MS Access; it sounds like it has similar limitations as far as scalability and thus it comes as no big surprise that it doesn’t measure up against Paradox, SQL Server, MySQL, etc.

You should try SqlLite - it is small and VERY fast. Not as robust as MySql, but it sounds like it may be better for what you are trying to do. Give it a try.

]]>