3GB executables

[Alvaro Cardoso]

Hi everyone,
I have a checkers playing program made with PBCC 2.1,
I have also 2.7GB of End_Game_Table_Bases that I would like to
load into memory for the fastest random access possible.
I red somewehere that for an executable to use up to 3GB of RAM
you need:

in the boot.ini:
[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(2)\WINNT
[operating systems]
multi(0)disk(0)rdisk(0)partition(2)\WINNT="Windows XP Professional" /3GB

you need also to mark the executable to be 3GB space aware by
making the IMAGE_FILE_LARGE_ADDRESS_AWARE bit set in it's
image header.
How can I set this bit in my PBCC21 .exe files?

Thanks in advance,
Alvaro Cardoso




------------------

[Paul Dixon]

Alvaro,
sounds like you want 2.7GB of DATA not EXE.

Paul.

------------------

[Adam J. Drake]

Alvaro,

The /3GB switch will cause you nothing but heartache on a 32 bit system. I tried it on mine and the machine wouldn't even boot.



------------------

[Marco Pontello]

Originally posted by Adam J. Drake:
I tried it on mine and the machine wouldn't even boot.

Maybe you tried on a Win 2000 (server) machine? Or XP + SP1?
It's known to cause crashes on that systems.

Bye!

------------------
Try Online TrID file identifier! Recognize over 1.900 filetypes and counting...
Give a powerfull scriptable Lua interface to your application with PowerBLua
Leakout - Firewall outbound control tester - BitmapRip Extract/recover bitmaps from any file!

[This message has been edited by Marco Pontello (edited April 11, 2006).]

[Hubert Brandel]

Hi,

Originally posted by Alvaro Cardoso:
I have a checkers playing program ... fastest random access possible.
I red somewehere that for an executable to use up to 3GB of RAM

If your program will need 3 GB + Windows 512 MB, your PC needs at
minimum 4 GB off RAM Hardware, otherwise it have to put ram from disk,
which is not fast. And who do you think will have that and modify its
WinXP for a simple game ?

So I think you should try to get a small exe and have the data in datafiles with
indexes - have a look at cheetah or build your own ...

Bye
Hubert


------------------

[Michael Mattias]

My question is, what the heck kind of data requires 3 Gb of storage - of any kind?

I think you could keep a play-by-play record of every checkers game played since the dawn of man in a lot less that that.

[Michael Mattias]

Wait a minute.

On Win/32, you can't access more than 2 Gb USER memory in any one process anyway.

[Adam J. Drake]

What Michael said...

Maybe you tried on a Win 2000 (server) machine? Or XP + SP1?
It's known to cause crashes on that systems.

I misspoke slightly - it would boot - but it didn't have enough resources to load my desktop settings. NOT a good switch to flip in a 32 bit system. I ended up having to boot through my Win XP CD to the recovery console to edit my INI file because it couldn't load a single app otherwise to do so. DO NOT DO THIS ON A 32 BIT SYSTEM

It was XP Pro SP2...On a 32 bit system it is an unwise move and was not well thought through by me. Some of the hardware guys I know seemed to think it was a good answer to speeding up the machine since it did have 4GB of RAM, alas, I will be taking no configuration tips regarding that type thing again...

------------------


[This message has been edited by Adam J. Drake (edited April 11, 2006).]

[Marco Pontello]

Originally posted by Adam J. Drake:
It was XP Pro SP2...On a 32 bit system it is an unwise move and was not well thought through by me. Some of the hardware guys I know seemed to think it was a good answer to speeding up the machine since it did have 4GB of RAM, alas, I will be taking no configuration tips regarding that type thing again...

OK, but having a 64bit CPU doesn't make any difference regarding this switch.
On 32bit OS the CPU is run in legacy mode, so it doesn't even know it have "more features" (64bit, 8 additional regs, etc.).
With 64bit OS (like the x64 edition of XP), the CPU is in "long" mode and the /3GB is superceeded by a /LARGEADDRESSAWARE that set a limit of 4GB virtual address space per 32-bit process.

The problem must have been somewhere else altough is not easy to say / guess where, these things always were a bit problematic.

Bye!

------------------
Try Online TrID file identifier! Recognize over 1.900 filetypes and counting...
Give a powerfull scriptable Lua interface to your application with PowerBLua
Leakout - Firewall outbound control tester - BitmapRip Extract/recover bitmaps from any file!

[This message has been edited by Marco Pontello (edited April 11, 2006).]

[Alvaro Cardoso]

My .exe is only 250k long, yes I need to load 2.7GB of data into memory.
EGTB are huge tables that have all positions of a certain piece count.
In my case it is an EGTB of 6 piece wich means I have the theoretical game result
of every thinkable position of 6 pieces or less. That gives tremendous strengh to
the program. Currentely I have 2GB, but I can add an extra GB of DDR400.
And yes, others have done it but with C. All they have to do was to set an option
in the compiler to set that bit in the image header.
If we need is to set a bit in the .exe file it should be easy,
Bu I don't know how to locate it.

Best regards,
Alvaro Cardoso



------------------

[Steven Pringels 3]

Hi Alvaro,

I think what you need is a good data structure and a binary tree
index mechanism and store the 3GB of data in a file which is indexed
some way.

Even if the program would be able to hold 3GB of data, the method
of accessing is what counts to get to resolve the theoretical game
result.

Since you bring up the notion of 'Tables' I guess a BTree would just
that what you need. Check out Cheetah, Tsunami, Patrice Terrier BTree,
and the new SlamDB.


------------------
I'd rather be hated for who I am, then being loved for who I am not.

[Michael Mattias]

All they have to do was to set an option in the compiler to set that bit in the image header.
If we need is to set a bit in the .exe file it should be easy,

No you can't just 'set a bit in the EXE file.'

Let's return to fundmentals...

>yes I need to load 2.7GB of data into memory

Can't be done on Win/32. Period. End of discussion. (OK, so you could memory-map PART of it). So it's time for Plan B.

Disk is memory, too. (As a matter of fact disk used be referred to regularly as 'secondary memory').

Assuming you really require that much space for your data, why can't you keep it on disk and go get the portion you need when you need it?

And as I pointed out earlier, I really doubt you need that much physical storage of any kind to retain the " theoretical game result of every thinkable position of 6 pieces or less."

I'm really weak on permutations and things like that; maybe someone else knows how to get the number of possibilities for 6 pieces each with 64 possible values." You only need one byte for each such possibility (one-half byte "piece id", one-half byte "position number").

[Paul Dixon]

Alvaro,
try the following 3 links. They explain what to do and when it'll work and it looks like the 3rd link provides you, at the bottom of the page, with the program file needed to modify your EXE file to take advantage of the extra RAM.

I've only recently smashed through the 256MB limit imposed by Win98 so I can't check this out to see if it works.

Paul.
http://winhlp.com/WxMoreThan2GB.htm
http://www.microsoft.com/whdc/system...AE/PAEmem.mspx
http://gid.cimne.upc.es/support/gid3gb/index.html


------------------

[Alvaro Cardoso]

Hurray Paul !!!
That's it.
Many many many thanks to you Paul Dixon.

To the ones that suggest using Btrees and the hard disk, I can say this:
In chess/checkers programming there are two functions: one that converts the board configuration
to an unique index (that we use for the lookup in the EGTB table), and
one that converts an index to the board position, this last one is
only used in the EGTB database generator that can take days or months
to compute. Chess programs use the hard disk with a clever/dedicated/special purpose
EGTB caching algorithm, still it kills performance and it can only
be used in the first 10-15 plies (depth) of the search tree.
The search tree grows exponentially from one ply to the next (depth).
So only a blazing fast in memory method is what I want for my particular
case because the program searches 600000 positions per second and the
slower it searches the weaker it gets.

Best regards,
Alvaro Cardoso




------------------

[Paul Dixon]

Alvaro,
glad you found what you want.

With such large tables, if you access them randomly you will almost always get a cache miss and end up waiting for data to be retrieved from main memory.
You say "a blazing fast in memory method is what I want" but did you realise that "blazing fast" main memory, although many times faster than hard disk, is actually still many times slower than the CPU?

Most people forget this (or perhaps never knew it) but it can take hundreds of CPU clock cycles (typically 500clks on my PC) to look up a single non-cached byte from main memory. It's often faster to calculate the data than to look it up in such a large table. You can do a LOT of work in 500clks.

Once accessed, the data is then in cache so the next time you need it you can get it instantly.
Using this fact you can sometimes save time by prefetching the next item of data long before you need it so that when it's required it will be in the cache ready.

Perhaps you already know this..

Paul.


------------------

[Michael Mattias]

Call me an old fuddy-duddy, but...

Were I to design an application, then add up the resource required to support that design and discover I 'needed' 2.7 Gb of RAM I think I'd tell myself, "Back to the drawing board, Michael!)

(That or apply for a job at the largest software company I could find in Redmond WA.)

[Paul Dixon]

Old Fuddy-Duddy,
I think Alvaro doesn't intend to sell this program to Joe Public but wants to use it in computer vs computer contests or computer vs human contests along the lines of Deep Blue vs Kasparov.
In these circumstances, 2.7GB lookup tables in RAM is a reasonable approach. It's a lot less hardware than IBM put up against Mr.Kasparov.

Paul.


------------------

[David J Walker]

http://users.ox.ac.uk/~uzdm0006/scans/1kchess/

------------------