Announcement

Collapse
No announcement yet.

3GB executables

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • 3GB executables

    Hi everyone,
    I have a checkers playing program made with PBCC 2.1,
    I have also 2.7GB of End_Game_Table_Bases that I would like to
    load into memory for the fastest random access possible.
    I red somewehere that for an executable to use up to 3GB of RAM
    you need:

    in the boot.ini:
    [boot loader]
    timeout=30
    default=multi(0)disk(0)rdisk(0)partition(2)\WINNT
    [operating systems]
    multi(0)disk(0)rdisk(0)partition(2)\WINNT="Windows XP Professional" /3GB

    you need also to mark the executable to be 3GB space aware by
    making the IMAGE_FILE_LARGE_ADDRESS_AWARE bit set in it's
    image header.
    How can I set this bit in my PBCC21 .exe files?

    Thanks in advance,
    Alvaro Cardoso




    ------------------

  • #2
    Alvaro,
    sounds like you want 2.7GB of DATA not EXE.

    Paul.

    ------------------

    Comment


    • #3
      Alvaro,

      The /3GB switch will cause you nothing but heartache on a 32 bit system. I tried it on mine and the machine wouldn't even boot.



      ------------------
      Adam Drake
      Drake Software

      Comment


      • #4
        Originally posted by Adam J. Drake:
        I tried it on mine and the machine wouldn't even boot.
        Maybe you tried on a Win 2000 (server) machine? Or XP + SP1?
        It's known to cause crashes on that systems.

        Bye!

        ------------------
        Try Online TrID file identifier! Recognize over 1.900 filetypes and counting...
        Give a powerfull scriptable Lua interface to your application with PowerBLua
        Leakout - Firewall outbound control tester - BitmapRip Extract/recover bitmaps from any file!

        [This message has been edited by Marco Pontello (edited April 11, 2006).]
        -- The universe tends toward maximum irony. Don't push it.

        File Extension Seeker - Metasearch engine for file extensions / file types
        Online TrID file identifier | TrIDLib - Identify thousands of file formats

        Comment


        • #5
          Hi,

          Originally posted by Alvaro Cardoso:
          I have a checkers playing program ... fastest random access possible.
          I red somewehere that for an executable to use up to 3GB of RAM
          If your program will need 3 GB + Windows 512 MB, your PC needs at
          minimum 4 GB off RAM Hardware, otherwise it have to put ram from disk,
          which is not fast. And who do you think will have that and modify its
          WinXP for a simple game ?

          So I think you should try to get a small exe and have the data in datafiles with
          indexes - have a look at cheetah or build your own ...

          Bye
          Hubert


          ------------------
          Regards,
          Hubert

          ------------------------------------
          http://familie-brandel.de/index_e.html

          Comment


          • #6
            My question is, what the heck kind of data requires 3 Gb of storage - of any kind?

            I think you could keep a play-by-play record of every checkers game played since the dawn of man in a lot less that that.

            Michael Mattias
            Tal Systems (retired)
            Port Washington WI USA
            [email protected]
            http://www.talsystems.com

            Comment


            • #7
              Wait a minute.

              On Win/32, you can't access more than 2 Gb USER memory in any one process anyway.

              Michael Mattias
              Tal Systems (retired)
              Port Washington WI USA
              [email protected]
              http://www.talsystems.com

              Comment


              • #8
                What Michael said...

                Maybe you tried on a Win 2000 (server) machine? Or XP + SP1?
                It's known to cause crashes on that systems.
                I misspoke slightly - it would boot - but it didn't have enough resources to load my desktop settings. NOT a good switch to flip in a 32 bit system. I ended up having to boot through my Win XP CD to the recovery console to edit my INI file because it couldn't load a single app otherwise to do so. DO NOT DO THIS ON A 32 BIT SYSTEM

                It was XP Pro SP2...On a 32 bit system it is an unwise move and was not well thought through by me. Some of the hardware guys I know seemed to think it was a good answer to speeding up the machine since it did have 4GB of RAM, alas, I will be taking no configuration tips regarding that type thing again...

                ------------------


                [This message has been edited by Adam J. Drake (edited April 11, 2006).]
                Adam Drake
                Drake Software

                Comment


                • #9
                  Originally posted by Adam J. Drake:
                  It was XP Pro SP2...On a 32 bit system it is an unwise move and was not well thought through by me. Some of the hardware guys I know seemed to think it was a good answer to speeding up the machine since it did have 4GB of RAM, alas, I will be taking no configuration tips regarding that type thing again...
                  OK, but having a 64bit CPU doesn't make any difference regarding this switch.
                  On 32bit OS the CPU is run in legacy mode, so it doesn't even know it have "more features" (64bit, 8 additional regs, etc.).
                  With 64bit OS (like the x64 edition of XP), the CPU is in "long" mode and the /3GB is superceeded by a /LARGEADDRESSAWARE that set a limit of 4GB virtual address space per 32-bit process.

                  The problem must have been somewhere else altough is not easy to say / guess where, these things always were a bit problematic.

                  Bye!

                  ------------------
                  Try Online TrID file identifier! Recognize over 1.900 filetypes and counting...
                  Give a powerfull scriptable Lua interface to your application with PowerBLua
                  Leakout - Firewall outbound control tester - BitmapRip Extract/recover bitmaps from any file!

                  [This message has been edited by Marco Pontello (edited April 11, 2006).]
                  -- The universe tends toward maximum irony. Don't push it.

                  File Extension Seeker - Metasearch engine for file extensions / file types
                  Online TrID file identifier | TrIDLib - Identify thousands of file formats

                  Comment


                  • #10
                    My .exe is only 250k long, yes I need to load 2.7GB of data into memory.
                    EGTB are huge tables that have all positions of a certain piece count.
                    In my case it is an EGTB of 6 piece wich means I have the theoretical game result
                    of every thinkable position of 6 pieces or less. That gives tremendous strengh to
                    the program. Currentely I have 2GB, but I can add an extra GB of DDR400.
                    And yes, others have done it but with C. All they have to do was to set an option
                    in the compiler to set that bit in the image header.
                    If we need is to set a bit in the .exe file it should be easy,
                    Bu I don't know how to locate it.

                    Best regards,
                    Alvaro Cardoso



                    ------------------

                    Comment


                    • #11
                      Hi Alvaro,

                      I think what you need is a good data structure and a binary tree
                      index mechanism and store the 3GB of data in a file which is indexed
                      some way.

                      Even if the program would be able to hold 3GB of data, the method
                      of accessing is what counts to get to resolve the theoretical game
                      result.

                      Since you bring up the notion of 'Tables' I guess a BTree would just
                      that what you need. Check out Cheetah, Tsunami, Patrice Terrier BTree,
                      and the new SlamDB.


                      ------------------
                      I'd rather be hated for who I am, then being loved for who I am not.
                      So here we are, this is the end.
                      But all that dies, is born again.
                      - From The Ashes (In This Moment)

                      Comment


                      • #12
                        All they have to do was to set an option in the compiler to set that bit in the image header.
                        If we need is to set a bit in the .exe file it should be easy,
                        No you can't just 'set a bit in the EXE file.'

                        Let's return to fundmentals...

                        >yes I need to load 2.7GB of data into memory

                        Can't be done on Win/32. Period. End of discussion. (OK, so you could memory-map PART of it). So it's time for Plan B.

                        Disk is memory, too. (As a matter of fact disk used be referred to regularly as 'secondary memory').

                        Assuming you really require that much space for your data, why can't you keep it on disk and go get the portion you need when you need it?

                        And as I pointed out earlier, I really doubt you need that much physical storage of any kind to retain the " theoretical game result of every thinkable position of 6 pieces or less."

                        I'm really weak on permutations and things like that; maybe someone else knows how to get the number of possibilities for 6 pieces each with 64 possible values." You only need one byte for each such possibility (one-half byte "piece id", one-half byte "position number").


                        Michael Mattias
                        Tal Systems (retired)
                        Port Washington WI USA
                        [email protected]
                        http://www.talsystems.com

                        Comment


                        • #13
                          Alvaro,
                          try the following 3 links. They explain what to do and when it'll work and it looks like the 3rd link provides you, at the bottom of the page, with the program file needed to modify your EXE file to take advantage of the extra RAM.

                          I've only recently smashed through the 256MB limit imposed by Win98 so I can't check this out to see if it works.

                          Paul.
                          http://winhlp.com/WxMoreThan2GB.htm
                          http://www.microsoft.com/whdc/system...AE/PAEmem.mspx
                          http://gid.cimne.upc.es/support/gid3gb/index.html


                          ------------------

                          Comment


                          • #14
                            Hurray Paul !!!
                            That's it.
                            Many many many thanks to you Paul Dixon.

                            To the ones that suggest using Btrees and the hard disk, I can say this:
                            In chess/checkers programming there are two functions: one that converts the board configuration
                            to an unique index (that we use for the lookup in the EGTB table), and
                            one that converts an index to the board position, this last one is
                            only used in the EGTB database generator that can take days or months
                            to compute. Chess programs use the hard disk with a clever/dedicated/special purpose
                            EGTB caching algorithm, still it kills performance and it can only
                            be used in the first 10-15 plies (depth) of the search tree.
                            The search tree grows exponentially from one ply to the next (depth).
                            So only a blazing fast in memory method is what I want for my particular
                            case because the program searches 600000 positions per second and the
                            slower it searches the weaker it gets.

                            Best regards,
                            Alvaro Cardoso




                            ------------------

                            Comment


                            • #15
                              Alvaro,
                              glad you found what you want.

                              With such large tables, if you access them randomly you will almost always get a cache miss and end up waiting for data to be retrieved from main memory.
                              You say "a blazing fast in memory method is what I want" but did you realise that "blazing fast" main memory, although many times faster than hard disk, is actually still many times slower than the CPU?

                              Most people forget this (or perhaps never knew it) but it can take hundreds of CPU clock cycles (typically 500clks on my PC) to look up a single non-cached byte from main memory. It's often faster to calculate the data than to look it up in such a large table. You can do a LOT of work in 500clks.

                              Once accessed, the data is then in cache so the next time you need it you can get it instantly.
                              Using this fact you can sometimes save time by prefetching the next item of data long before you need it so that when it's required it will be in the cache ready.

                              Perhaps you already know this..

                              Paul.


                              ------------------

                              Comment


                              • #16
                                Call me an old fuddy-duddy, but...

                                Were I to design an application, then add up the resource required to support that design and discover I 'needed' 2.7 Gb of RAM I think I'd tell myself, "Back to the drawing board, Michael!)

                                (That or apply for a job at the largest software company I could find in Redmond WA.)

                                Michael Mattias
                                Tal Systems (retired)
                                Port Washington WI USA
                                [email protected]
                                http://www.talsystems.com

                                Comment


                                • #17
                                  Old Fuddy-Duddy,
                                  I think Alvaro doesn't intend to sell this program to Joe Public but wants to use it in computer vs computer contests or computer vs human contests along the lines of Deep Blue vs Kasparov.
                                  In these circumstances, 2.7GB lookup tables in RAM is a reasonable approach. It's a lot less hardware than IBM put up against Mr.Kasparov.

                                  Paul.


                                  ------------------

                                  Comment


                                  • #18
                                    http://users.ox.ac.uk/~uzdm0006/scans/1kchess/

                                    ------------------

                                    Comment

                                    Working...
                                    X