Full Unicode/Ansi-Support in /testing branch

The alpha/beta tester's forum for ZeosLib 7.0.x series

Report problems concerning our Delphi 2009+ version and new Zeoslib 7.0 features here.

This is a forum that will be removed once the 7.X version goes into stable!!

Moderators: gto, EgonHugeist, olehs

User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

miab3,

hope i got it now R2006.

Question, Michal. What do you think would be a nice RowSize if i address all Strings and Bytes as Pointer?

on 64Bit OS a Pointer has 8Bytes on 32Bit a pointer has 4 Bytes. The biggest values currently (on my working copy) is either stBytes (Pointer+Smallint(Length)) or Extendend(10Bytes).

an 8KB Row could pick up 1024/2048 columns with Data(x) on 64Bit/32Bit and so on.
That's the first step for full Mem-optimazion i want to have. Currently i've no idea how to get a varying size of the RowAccessor running without speed decrease. But moving the strings out of that array and only address reserved memory is equal fast but eats only the Memory for the strings we need (except the ByteArray 'em selves).

I'll test it on 64Bit now if it's equal stable..

So what do you think are 4KB enough or should i prefere 8KB/row? In the current stage i could also add defines bud they are ugly..
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
miab3
Zeos Test Team
Zeos Test Team
Posts: 1305
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

EgonHugeist,

After R2006 Lazarus for Linux compiles without errors.

On the transition to pointers:
- Whether it is really will be faster?
- In accordance with the standards of db servers need more, such as 4096 fields for MySQL:
http://database-management-systems.find ... d-vs-MySQL
- I am using 128kB buffer without losing speed.

Michal
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

miab3,
are there such limitations for MsSQL, ADO, Oracle? I don't know it yet.
I am using 128kB buffer without losing speed.
Well this i know, Michal. Because you propsed this patch before.
But that means: 1.000.000 Rows * 128KB = ~122GB Memory. Which means the OS is starting moving memory to the Harddisk (if so) and this will slow down the performance.

I have a lightning fast Laptop with 32Gb but such an select would kill all performance. Now we've protocols like SQLite or MySQL which are fetching all data once (libmysql does it, then you got your result).

My intention is/was to store only pointers to strings or byte arrys. The target pointer contains allocated Memory. No string no memory. Long/very large strings more memory. Equal to Lob-Streams. Actually we've a patch inside the sources which says: (MySQLUtils.pas)

Code: Select all

  
{ Fix by the HeidiSql team. - See their SVN repository rev.775 and 900}
  { SHOW FULL PROCESSLIST on 4.x servers can return veeery long FIELD_TYPE_VAR_STRINGs.
  The following helps avoid excessive row buffer allocation later on. }
  if (Result = stString) and (PlainDriver.GetFieldLength(FieldHandle) > 8192) then
     Result := stAsciiStream;

  if (Result = stUnicodeString) and (PlainDriver.GetFieldLength(FieldHandle) > 8192) then
     Result := stUnicodeStream;
Such a limitation was made to avoid the RowbufferWidthExceeded exception. What would happen on selecting 20 of such field? They won't fit into your/our RowBuffer. But that can simply be done on using only Memory addresses like Streams are doing this.

Exacly this i've ready now (works on Win64 too). Remainig would be a varying size instead of a fixed (currently 64Kb on my side 128 on yours) of the RowBuffer to handle x Fields and the memory which sould be used.

Maybe i commit first if the last goal was also solved?
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
miab3
Zeos Test Team
Zeos Test Team
Posts: 1305
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

EgonHugeist,

Have you tried?
This is only the maximum size of the buffer that virtually no change in the size of consumed memory and speed.

Michal
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

miab3,

nope i didn't test it. But i can read the souces and each fetched row pointer of an virtual or cached resultset will be added to a TList. Then we can navigate between the columns and it doesn't matter if the provider supports forward-only fetching or whatever.

Did you execute TZQuery.Last or TZQuery.Recordcount if you've select a very huge count of rows + columns?

Well but i will test it also by my selves. IT would be interesting if we've an optimation or not.. On the other hand: never change a running system (((: and i'll trash the approach.
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
patyit
Junior Boarder
Junior Boarder
Posts: 37
Joined: 10.02.2012, 18:39

Post by patyit »

Hi !

I'm finally finishes the project on Linux (xubuntu), now I going to do same on Win7,
but got an error while installing on Lazarus:

C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(384,35) Error: Identifier not found "MultiByteToWideChar"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(387,30) Error: Identifier not found "MultiByteToWideChar"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(400,35) Error: Identifier not found "MultiByteToWideChar"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(400,54) Error: Identifier not found "MB_PRECOMPOSED"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(403,30) Error: Identifier not found "MultiByteToWideChar"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(403,49) Error: Identifier not found "MB_PRECOMPOSED"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(438,35) Error: Identifier not found "WideCharToMultiByte"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(441,30) Error: Identifier not found "WideCharToMultiByte"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(456,35) Error: Identifier not found "WideCharToMultiByte"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(457,31) Error: Identifier not found "WC_COMPOSITECHECK"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(457,47) Error: Identifier not found "WC_DISCARDNS"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(457,62) Error: Identifier not found "WC_SEPCHARS"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(457,79) Error: Identifier not found "WC_DEFAULTCHAR"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(461,30) Error: Identifier not found "WideCharToMultiByte"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(462,31) Error: Identifier not found "WC_COMPOSITECHECK"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(462,47) Error: Identifier not found "WC_DISCARDNS"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(462,62) Error: Identifier not found "WC_SEPCHARS"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(462,79) Error: Identifier not found "WC_DEFAULTCHAR"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(640,21) Error: Identifier not found "GetACP"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(659,33) Hint: Mixing signed expressions and longwords gives a 64bit result
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(902) Fatal: There were 19 errors compiling module, stopping

Lazarus 1.1 svn, Fpc 2.6.1 svn, Win7 Starter. (with FPCUP), Zeos7 svn testing from now.
Last week I can't remember the rev.no. (below 2000) was ok.
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

patyit,

hmm their is the unit windows missing. Did compile and test it with fpc2.6.0/LCL1.02 today. No issues to see. Current rev: 2007 can you please check it again? If the issue remains where can i download an FPC2.6.1?
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
patyit
Junior Boarder
Junior Boarder
Posts: 37
Joined: 10.02.2012, 18:39

Post by patyit »

EgonHugeist,

You are right, windows unit is //-ed by me someday when I have problems with compiling Lazarus on Linux ... and normally forget it ...

Sorry for that ! It is now all right.
(tested with rev. 2008 on Win7 Starter and XUbuntu 12.04 32bit)

By the way the easiest way to install latest Fpc and Lazarus is usung FPCUP utility from :
https://bitbucket.org/reiniero/fpcup/downloads
miab3
Zeos Test Team
Zeos Test Team
Posts: 1305
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

EgonHugeist,

Zeos7.0.1-beta branches_ testing_R2010:
http://svn.code.sf.net/p/zeoslib/code-0 ... s/testing/
compiles and runs on (I have tested):

- D2006,
- D2007 -> C++,
- DXE2 32/64 -> C++(32),
- Lazarus Win 1.1/fpc 2.7.1 32/64,
- Lazarus Lin 1.1/fpc 2.7.1 64 on Debian 64-bit (LMDE 201204).

(Firebird 2.5.3, MySQL 5.5.18, MariaDB 5.5.28, Oracle 11g EE, PostgreSQL 9.2.1, MSSQL 2008 R2 Express)

Michal
Zoran
Senior Boarder
Senior Boarder
Posts: 55
Joined: 07.05.2010, 22:32

Post by Zoran »

EgonHugeist wrote:
On non Windows plattforms do the functions i call not exsits. Thats why i've added the LConvEncoding unit from Lazarus. But actually we are not allowed to provide this unit(which would happen without Lazarus) so i've added the functionality with the WITH_LCONVENCODING define in ZeoaLazarus.inc. (we need this unit only on non MsWindows plattforms)

So uncomment:

{.$IFNDEF MSWINDOWS}
{.$DEFINE WITH_LCONVENCODING} //Only available for Lazarus to activate the LConv uncomment this define
{.$ENDIF}
I believe that you can use {$IFDEF LCL} to make difference between Lazarus (LCL dependency) and FPC without Lazarus (no LCL dependency).

Then we know that LConvEncoding unit is here (it is part of LCL), as the project has LCL dependecy, so we don't have to include it with Zeos.

So, instead of commenting defines, here I would recommend something like this:

Code: Select all

{$IFDEF FPC}{$IFDEF LCL}
  {$DEFINE WITH_LCONVENCODING}
{$ENDIF}{$ENDIF}
And generally, as LCL controls use UTF8, and FPC by itself does not, you often need to distinguish FPC projects which have LCL dependency (that is, Lazarus projects) from those which don't have it. You can ask for LCL define.
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

patyit, miab3,

thanks for confirmation!

Zoran,

uh this sounds great! I'll test it right now. If you are right (i think so) then i know some more places where such an define can bea helpfull.

Question: Is this define provided by FPC or an build in define on compiling the Project?
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
Zoran
Senior Boarder
Senior Boarder
Posts: 55
Joined: 07.05.2010, 22:32

Post by Zoran »

EgonHugeist wrote: Question: Is this define provided by FPC or an build in define on compiling the Project?
It is mentioned in Lazarus wiki - http://wiki.lazarus.freepascal.org/Code ... .2F_macros

I don't know where it is defined, but I tested to see if it is defined in a Lazarus application.

The test, from Lazarus main menu chose:

1. chose Project->New Project->Application
add two lines to test fpc and lcl defines, you can put it at the beginning of lpr file:

Code: Select all

program Project1;

{$mode objfpc}{$H+}

uses
  {$IFDEF UNIX}{$IFDEF UseCThreads}
  cthreads,
  {$ENDIF}{$ENDIF}
  Interfaces, // this includes the LCL widgetset
  Forms, Unit1
  { you can add units after this };

{$R *.res}

begin
  {$ifdef fpc}writeln('yes FPC');{$else}writeln('NO FPC');{$endif}
  {$ifdef lcl}writeln('yes LCL');{$else}writeln('NO LCL');{$endif}

  RequireDerivedFormResource := True;
  Application.Initialize;
  Application.CreateForm(TForm1, Form1);
  Application.Run;
end.
This is the output:
yes FPC
yes LCL
2. Now chose Project->New Project->Program
add the same two lines in program, now we have:

Code: Select all

program Project1;

{$mode objfpc}{$H+}

uses
  {$IFDEF UNIX}{$IFDEF UseCThreads}
  cthreads,
  {$ENDIF}{$ENDIF}
  Classes
  { you can add units after this };

begin
  {$ifdef fpc}writeln('yes FPC');{$else}writeln('NO FPC');{$endif}
  {$ifdef lcl}writeln('yes LCL');{$else}writeln('NO LCL');{$endif}
end.
Now, the output is this:
yes FPC
NO LCL
So, the LCL projects do have this LCL define, non-LCL projects do not have it. Therefore, I think that it is just what we need.
User avatar
EgonHugeist
Zeos Project Manager
Zeos Project Manager
Posts: 1936
Joined: 31.03.2011, 22:38

Post by EgonHugeist »

Zoran,

thanks for the details. Yes i found it too: http://wiki.freepascal.org/Code_Conversion_Guide

So i'll start some rework. BTW: http://svn.code.sf.net/p/zeoslib/code-0 ... esting-7.1 has been started. If you doubt... ((:
Best regards, Michael

You want to help? http://zeoslib.sourceforge.net/viewtopic.php?f=4&t=3671
You found a (possible) bug? Use the new bugtracker dude! http://sourceforge.net/p/zeoslib/tickets/

Image
miab3
Zeos Test Team
Zeos Test Team
Posts: 1305
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

EgonHugeist,

Zeos7.1-beta branches_ testing_R2018:
http://svn.code.sf.net/p/zeoslib/code-0 ... sting-7.1/

In ZAbstractConnection.pas line 335

Unnecessary (disturbing):

FUTF8StringAsWideField := False;

Michal
miab3
Zeos Test Team
Zeos Test Team
Posts: 1305
Joined: 11.05.2012, 12:32
Location: Poland

Post by miab3 »

EgonHugeist,

Michael,
Is it possible to speed up the update/modify SQLite with the new CAPI?
How?

Michal
Locked