Announcement

Collapse
No announcement yet.

Very unusual problem.

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Very unusual problem.

    I have been adding additional data to a help engine in the form of resource$ statements and have just found what looks like a limitation in how much memory the resource$ statement can handle. About 25% of the time that engine just exits with no explanation. I set up the following,
    Code:
      TRACE NEW "trace.txt"
      TRACE ON
    And stepped through I tracked where the app either crashed or displayed an error message.
    Code:
        lbl1:
    
        avxseg$      = resource$(RCDATA, 2022)      ' <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    
        MyEr$ = last_error_string
    
        MsgBox MyEr$
    
        lbl2:
    The resource$ notation is correct but when running the OS based LastError call, I get this error.
    Code:
        Not enough memory resources are available to process this command.
    The actual amount of data is about 350k of RTF format text and this seems to be the problem.

    If the app does not crash, the text that the resource$ line activates works correctly and displays the RTF but at about 25% of the runs of the engine, it crashes trying to execute this line of code.

    Has anyone ever seen this problem before ? I would be interested if anyone has solved this problem in their own work.
    hutch at movsd dot com
    The MASM Forum

    www.masm32.com

  • #2
    Steve,
    #RESOURCE RCDATA, ResID, "filespec.DAT"
    Returns predefined
    resource data.
    Syntax r$ = RESOURCE$(RCDATA, ResID)
    You are still using Windows 7, right?

    Comment


    • #3
      I found it, a duplicate entry with a non existent resource ID. Useful enough, I got some practice using the PB trace and isolated the line with labels.

      Jim, I have been using Win10 64 for about 5 years. I do have a nice Win7 64 box but don't use it much.
      hutch at movsd dot com
      The MASM Forum

      www.masm32.com

      Comment


      • #4
        Glad you found it.

        Comment


        • #5
          .. stepped through I tracked where the app either crashed or displayed an error message.
          In general you need to be real careful and NOT assume that a program 'crashes' because of an error at that point. The actual cause of the crash may be an error hundreds or thousands of lines of code (source or object) from that "death point."

          Using the stepping debugger to "stop on untrapped error" is probably the closest you can come to finding the actual point of error... but bear in mind external calls (e.g to the WinAPI) will NOT trip this setting.
          Michael Mattias
          Tal Systems Inc. (retired)
          Racine WI USA
          [email protected]
          http://www.talsystems.com

          Comment


          • #6
            Code:
            lbl1:
            avxseg$ = resource$(RCDATA, 2022) ' <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
            MyEr$ = last_error_string
            MsgBox MyEr$
            lbl2:
            The resource$ notation is correct but when running the OS based LastError call, I get this error

            .
            You don't mean the Widows API call "GetLastError(), do you? That cannot reliably be used if mixed in with PowerBASIC intrinsics, especially for anything involving a string.

            FWIW, PB should* return some kind of ERR on the RESOURCE$() call; I don't see your code reading that, nor is any active error trap ('ON ERROR'... statement) shown.

            For that matter, if you built your executable with duplicate IDs for type RCDATA resources using '#RESOURCE' statements, that should* have been caught at compile time.

            MCM
            * should does not mean "does" .. I have found a number of PB intrinsics which don't always return a PB error.
            Michael Mattias
            Tal Systems Inc. (retired)
            Racine WI USA
            [email protected]
            http://www.talsystems.com

            Comment


            • #7
              Hi Michael,

              Its not like it matters as I found the line where the error occurred and GetLastError() indicated that it was a memory page fault so I had isolated the line and had at least some idea of what was wrong with it. It just happened the the index for the entry was out of place which was a left over of a mod I did some months ago and the reason why the link (in the app ) worked (at times) was the line further down the long list that gave a valid link. As the first link line was the error, just deleting it allowed the valid link to work correctly.

              The app is a help file engine and while I have compacted the number of location for each entry, I still have to put in 4 entries to get a new page working properly. The app displays RTF pages with cross links to other pages.
              hutch at movsd dot com
              The MASM Forum

              www.masm32.com

              Comment


              • #8
                Originally posted by Michael Mattias View Post
                For that matter, if you built your executable with duplicate IDs for type RCDATA resources using '#RESOURCE' statements, that should* have been caught at compile time.

                MCM
                * should does not mean "does" .. I have found a number of PB intrinsics which don't always return a PB error.
                FWIW: If there are duplicate Resource name/ID for the same type of resource, both resources get compiled, either one may be accessed by ordinal number but only the first by ID. If you have two different types of resources with the same name/id both may be accessed by name/id.


                Comment


                • #9
                  What surprised me was that it ran when it tried to write to memory that was not allocated by the app. Normally you get a GP fault with an error of that type. It was a pain to isolate because of that. Once I found the line by narrowing down where the error occurred, the rest was reasonably straight forward. Some of the innards of Windows still fascinates me.
                  hutch at movsd dot com
                  The MASM Forum

                  www.masm32.com

                  Comment


                  • #10
                    FWIW: If there are duplicate Resource name/ID for the same type of resource, both resources get compiled, either one may be accessed by ordinal number but only the first by ID. If you have two different types of resources with the same name/id both may be accessed by name/id.
                    Not that it matters because I would NEVER (on purpose) create two resources of same type with same ID but...

                    How do you access a resource by ordinal? I cannot find any WinAPI functions which do that. I only have reference thru Win/XP here so maybe there is a new function(?)

                    MCM
                    Michael Mattias
                    Tal Systems Inc. (retired)
                    Racine WI USA
                    [email protected]
                    http://www.talsystems.com

                    Comment


                    • #11
                      The actual amount of data is about 350k of RTF format text and this seems to be the problem.
                      By the way, what would be the maximum amount of data that can be stored in a resource? I believe it is around 2GB ?

                      Comment


                      • #12
                        Originally posted by Michael Mattias View Post

                        Not that it matters because I would NEVER (on purpose) create two resources of same type with same ID but...

                        How do you access a resource by ordinal? I cannot find any WinAPI functions which do that. I only have reference thru Win/XP here so maybe there is a new function(?)

                        MCM
                        One way is to use ResourceHacker

                        Within your application, just use Loadmage or whatever. In the case of a bitmap/icon/cursor:

                        If the image resource is to be loaded by ordinal from the module, use the MAKEINTRESOURCE macro to convert the image ordinal into a form that can be passed to the LoadImage function.

                        Comment


                        • #13
                          Originally posted by Tim Lakinir View Post

                          By the way, what would be the maximum amount of data that can be stored in a resource? I believe it is around 2GB ?
                          See https://forum.powerbasic.com/forum/u...mit#post771041

                          Comment


                          • #14
                            If you have some reason for massive resources, putting them into DLLs would spread the load so it is more manageable for the calling executable. You will still run into absolute memory limits but if you are handling resources that big collectively you would load the start address of each DLL and dynamically load each one as you need it and free it after.
                            hutch at movsd dot com
                            The MASM Forum

                            www.masm32.com

                            Comment


                            • #15
                              Thanks to all, except that in https://forum.powerbasic.com/forum/u...454#post771454

                              Under Win/32, the number of processes which may each use 2 GB memory is limited only by the total available VIRTUAL memory on the system, which is the amount of installed RAM plus the max size of the swap file.
                              I have seen somewhere in the forum, some members uses #STACK 4194304 statement
                              does this stack statement would be able to kinda stabilize the available virtual memory to load a large file from the resource, see

                              https://forum.powerbasic.com/forum/u...645#post796645

                              Comment


                              • #16
                                Originally posted by Tim Lakinir View Post
                                I have seen somewhere in the forum, some members uses #STACK 4194304 statement
                                does this stack statement would be able to kinda stabilize the available virtual memory to load a large file from the resource
                                That just makes the stack 4MB instead of the default 1MB. Nothing to do with the HEAP .

                                You may be able to get a larger file with #OPTION LARGEMEM32 directive:
                                "For 32-bit Windows applications, this option sets the "Large Memory Model" flag. This allows your application to use more than the original limit of 2 Gigabytes of memory. Depending upon the version of Windows in use, and the installed memory, the exact increase may vary from computer to computer. In most cases, you will likely be limited to a total of approximately 3 Gigabytes".


                                Comment


                                • #17
                                  Tim,

                                  Stack memory is assigned to an executable file, default in most is 1 meg but 2 or 4 meg are viable. You tend to increase the stack allocation if you are using recursion or run something that allocates a massive amount of LOCAL variables. Under Win32 on a 32 bit system, you can allocate collectively just under 2 gigabytes and if you use #option largemem32 you get another block of memory, 7 to 800 meg to operate on the first big allocation. On a 64 bit system you get a bit more with a 32 bit app. Virtual memory (IE: Disk emulation of memory) is used by the OS to swap out some apps memory so that more can be allocated in dynamic memory (RAM).
                                  hutch at movsd dot com
                                  The MASM Forum

                                  www.masm32.com

                                  Comment


                                  • #18
                                    RE
                                    If you have some reason for massive resources, putting them into DLLs would spread the load so it is more manageable for the calling executable. You will still run into absolute memory limits but if you are handling resources that big collectively you would load the start address of each DLL and dynamically load each one as you need it and free it after.
                                    Also useful if you want to change other program data such as pictures, text or whatever without having to recompile - and re-test - the program itself. The "resource only" DLL makes for a nice maintenance tool IMO.

                                    MCM

                                    Michael Mattias
                                    Tal Systems Inc. (retired)
                                    Racine WI USA
                                    [email protected]
                                    http://www.talsystems.com

                                    Comment


                                    • #19
                                      Thank you Everyone

                                      One more question related to Steve's response

                                      If you have some reason for massive resources, putting them into DLLs would spread the load so it is more manageable for the calling executable.
                                      Does this means that instead of executing one big program, you can have one main program MainProg.exe calling several DLLs
                                      say child1.dll , child2.dll and child3.dll each of which would load a portion of the original big program into memory.

                                      Doing this way, it would be more efiicient in terms of memory allocation than having to execute one big program ?

                                      The question is that :
                                      If we have called say child2.dll and then close it and then go back to the MainProg.exe , will the memory of child2.dll
                                      still persist after we have close up child2.dll ?

                                      Are there additional memory clearance statements kinda similar to RESET and ERASE which we can use so that variables used in the DLLs are cleared
                                      when we exit these DLLs ?

                                      Comment


                                      • #20
                                        Originally posted by Tim Lakinir View Post
                                        Thank you Everyone

                                        One more question related to Steve's response



                                        Does this means that instead of executing one big program, you can have one main program MainProg.exe calling several DLLs
                                        say child1.dll , child2.dll and child3.dll each of which would load a portion of the original big program into memory.

                                        Doing this way, it would be more efiicient in terms of memory allocation than having to execute one big program ?

                                        The question is that :
                                        If we have called say child2.dll and then close it and then go back to the MainProg.exe , will the memory of child2.dll
                                        still persist after we have close up child2.dll ?

                                        Are there additional memory clearance statements kinda similar to RESET and ERASE which we can use so that variables used in the DLLs are cleared
                                        when we exit these DLLs ?
                                        That doesn't work that way. You can't 'close' a DLL. you may unload it from memory if you're using implicit linking through loadlibrary and a lot of hocus pocus with getting the pointer to each of the exposed functions. To invoke those implicit functions then you should use "Call Dword ..... using .... "

                                        In normal circumstances when explicit linking is used, DLL's stay linked to the process it was executed with, in this case MainProg.exe. The only benefit DLLs have is maintainability if patches need to be distributed only DLLs need to be downloaded if the main executable is kept as small as possible and another use-case is re-using functions for other executables. Download a program called Dependency Walker and see for yourself what DLLs an executable depends upon when launched.
                                        So here we are, this is the end.
                                        But all that dies, is born again.
                                        - From The Ashes (In This Moment)

                                        Comment

                                        Working...
                                        X