Announcement

Collapse
No announcement yet.

#Resource RcData Size Limit?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Michael Mattias
    replied
    I expect the absolulte number to be higher if you reboot your system and do not start much programms.
    Under Win/32, the number of processes which may each use 2 GB memory is limited only by the total available VIRTUAL memory on the system, which is the amount of installed RAM plus the max size of the swap file.

    And if you exceed this, it does not slow down, it just stops working unless you make sure your code is handling Error 7 (insufficient memory) . Been there, done that, got the T-shirt. (in my case it was a user who had installed his system with a relatively small swap file and did not allow Windows to manage the size of same).

    MCM

    Leave a comment:


  • Theo Gottwald
    replied
    In my Tests an executable has itself as limit of 2 GB in Size. Now the same limit may apply for Memory allocation under 32 bit programs.
    Therefore depending on the fragementation situation in Memory such an error can easily happen at this size - using 32 bit programs.
    Maybe there was no memory chunk (in one piece) large enough for more available.
    I expect the absolulte number to be higher if you reboot your system and do not start much programms.

    Leave a comment:


  • Dave Biggs
    replied
    That's what I found on Win 7 too.
    Yep. Further tests seem to show that the limit for compiling post #19 with Windows 7 (on my PC) is something like 930MB.

    Leave a comment:


  • Stuart McLachlan
    replied
    Originally posted by Dave Biggs View Post

    Did some more testing - using the #Resource RCDATA, RESID, "ResData.zip" method - no pre compiling to RES / external resource file.
    In Windows 10 things go awry on my PC when the .zip file exceeds 660MB or so.

    Above the limit, a small .exe file (~13k) is written to disk but PBWin.exe sits in memory eating up 25% of my CPU until killed with Task Manager's 'End Task'.
    That's what I found on Win 7 too.

    Leave a comment:


  • Dave Biggs
    replied

    Did some more testing - using the #Resource RCDATA, RESID, "ResData.zip" method - no pre compiling to RES / external resource file.
    In Windows 10 things go awry on my PC when the .zip file exceeds 660MB or so.

    Above the limit, a small .exe file (~13k) is written to disk but PBWin.exe sits in memory eating up 25% of my CPU until killed with Task Manager's 'End Task'.

    Code:
    #Compile Exe
    #Dim All
    #Include "win32api.inc"
    '#Resource Res, "RCDataZip.res"
    #RESOURCE RCDATA, RESID, "ResData.zip"
    
    Function MakeFileFromRes(sPath As String, ResId As String) As Long
     Local datasize, filehandle As Long
     Local buffer As String
    
        buffer   = Resource$(RcData, ResID)
        datasize = Len(buffer)
        ? ResID,,"ResID"+Str$(datasize)
    
       If datasize > 0 Then
         winbeep 800, 200
          filehandle = FreeFile
          Open spath For  Binary  As filehandle
            Put$ filehandle, buffer
          Close filehandle
          Function = %TRUE
       Else
          Function = %false
       End If
    End Function
    '------------------/MakeFileFromRes
    
    Function PBMain () As Long
     Local lRes As Long
    
      lRes = MakeFileFromRes("RCDataExtracted.zip", "RESID")
      ? "Done",,Str$(lRes)
    
    End Function
    '------------------/PBMain

    Leave a comment:


  • Rodney Hicks
    replied
    Right, Dave. I've been spoiled, never gave thought to the multi step process.All is good here.

    Leave a comment:


  • Dave Biggs
    replied
    Hi Rod,
    You need to create a .RC file using the line inside the #IF block. Then compile it with RC.EXE to generate the .RES file.

    (Otherwise..)
    Error 478 in C:\PBWin10\Samples\RCData\RCDataExe.bas(4:001): Resource file error
    Line 4: #Resource Res, "RCDataZip.res"

    Leave a comment:


  • Pierre Bellisle
    replied
    Using the method above, I did a manual compile under Windows 10.
    pbRes failed to create a valid .pbr from a valid 760mb .res file. The result was a 100 bytes file.
    Same procedure worked fine under Windows 7 though. [Added: Windows 8.1 worked fine also.]
    Last edited by Pierre Bellisle; 23 Mar 2018, 10:12 PM.

    Leave a comment:


  • Rodney Hicks
    replied
    Dave, I don't have a Win 10 system but I couldn't get your code to run on Win 7 with my 435,000+kb file. I did change #RESOURCE RES to #RESOURCE RCDATA, 1000, filename and changed "lRes = MakeFileFromRes("RCDataExtracted.zip", "RESID")" to "lRes = MakeFileFromRes("RCDataExtracted.zip", "#1000")" and it ran like a charm with the #IF block commented out. I get the Error 478 resource file error with your code on my machine.

    Leave a comment:


  • Pierre Bellisle
    replied
    Hey Gary,
    What if you try manually?

    REM Create a .res
    X:\Basic\PB-Compiler\bin\rc.exe "X:\Basic\Bas\SRC\MyFile.rc"

    REM Create a .pbr
    X:\Basic\PB-Compiler\bin\pbRes.exe "X:\Basic\Bas\SRC\MyFile.res"

    REM In the bas file have
    #RESOURCE "MyFile.pbr"

    REM Create a .exe
    X:\Basic\PB-Compiler\bin\pbWin.exe "X:\Basic\Bas\SRC\MyFile.bas" /L /I"X:\Basic\PB-Compiler\\WinApi"

    Leave a comment:


  • Dave Biggs
    replied
    Hi Rod,
    This is some test code I used. Runs OK in Windows 7 but I get the same failed .exe when compiled in WIndows 10 as Gary sees..!
    (Though my failed .exe is only 13K)

    Code:
    #Compile Exe
    #Dim All
    #Include "win32api.inc"
    #Resource Res, "RCDataZip.res"
    
    Function MakeFileFromRes(sPath As String, ResId As String) As Long
     Local datasize, filehandle As Long
     Local buffer As String
    
        buffer   = Resource$(RcData, ResID)
        datasize = Len(buffer)
        ? ResID,,"ResID"+Str$(datasize)
    
       If datasize > 0 Then
         winbeep 800, 200
          filehandle = FreeFile
          Open spath For  Binary  As filehandle
            Put$ filehandle, buffer
          Close filehandle
          Function = %TRUE
       Else
          Function = %false
       End If
    End Function
    '------------------/MakeFileFromRes
    
    Function PBMain () As Long
     Local lRes As Long
    
      lRes = MakeFileFromRes("RCDataExtracted.zip", "RESID")
      ? "Done",,Str$(lRes)
    
    End Function
    '------------------/PBMain
    
    #If 0
    //rc file contents - saved As RCDataZip.rc
    
    ResId RcData "ResData.zip"     // 835+MB archive. Compiled To .\RCDataZip.res
    #EndIf
    explorer listing

    Leave a comment:


  • Rodney Hicks
    replied
    Gary, could you post all the code from the offending program before any functions or procedures?

    Leave a comment:


  • Gary Beene
    replied
    But, Help does say that #Resource supports this syntax ...

    Code:
    #RESOURCE RES, "filespec.RES"
    But when I use that syntax with the 785MB RES file that resulted from my compilation as mentioned above, the result is a 77KB EXE file.




    Leave a comment:


  • Gary Beene
    replied
    Hmm.... it's been a while since I created a PBR file. So, I loaded this one-line .RC file and compiled it (BigFile.zip is 785MB) ...

    Code:
    123  RCDATA  "BigFile.zip"
    and, with the compiler set to "Create a .PBR file when compiling a .RC files", the compiler created a 785MB RES file, but a PBR file of only 100 bytes. I don't think that's how it's supposed to work!

    From Help ...

    When the "Create a .PBR when compiling .RC files" option on the compiler options tab is selected the IDE will also produce a .PBR file from the .RES file. This is achieved in one simple step: simply load the .RC file into the IDE and select Compile.

    Leave a comment:


  • Bob Carver
    replied
    I tested it with a 1.5GB iso file. PBWin 10.03 reads the entire file into memory in ~2MB chunks, and then exits with

    Error 478 in c:\Test\PB\test.bas(4:001): Resource file error
    Line 4: #RESOURCE RcData, 123, "H:\os\6001.18000.080118-1840_x86fre_Server_en-
    us-KRMSFRE_EN_DVD.iso"

    It didn't take half an hour for me though, took 90 seconds. A 550MB file actually produced an exe, but required 1.2GB of memory, in 42 seconds.

    Since it reads the specified file completely into memory, PBWin would seem to need to be able to allocate 2x the size of the file + have space for all the Windows dlls and whatever else in the 2GB of address space as (my version at least) isn't LargeAddressAware.

    I'm obviously guessing, but there's probably no hard maximum file size where the compiler outright rejects your rcdata files, but implementation wise, it seems to roughly be around 900MB-ish depending on how much code is generated, what other resources there are, etc.

    Leave a comment:


  • Stuart McLachlan
    replied
    Originally posted by Pierre Bellisle View Post
    Gary,

    You are using Windows 10, try to compile under Windows 7 if possible, if it works then it is a PowerBASIC / Windows 10 issue.

    I'm sure that if you use an external .rc file under any OS, it will work all-wight and the ~2gb limit will prevail .
    My test which I abandoned after 1/2 an hour or so, was under Win 7.

    Leave a comment:


  • Pierre Bellisle
    replied
    Gary,

    You are using Windows 10, try to compile under Windows 7 if possible, if it works then it is a PowerBASIC / Windows 10 issue.

    I'm sure that if you use an external .rc file under any OS, it will work all-wight and the ~2gb limit will prevail .

    Leave a comment:


  • Gary Beene
    replied
    Hi Pierre,

    I did this, where bigfile.zip is 785MB ...

    Code:
    #Resource RcData, 1000, "bigfile.zip"
    And, Stuart,
    Yes, I finally got the compiler to start cranking ... but it chugged along for quite a while before stopping/crashing with no error message.

    And, MCM,
    I've no problem going to another approach, but would like to know why #Resource approach doesn't work and perhaps know what the actual limitation is.

    Leave a comment:


  • Michael Mattias
    replied
    was trying to put a 785MB file into an EXE as a resource,
    Why? Do you really need to do that? (I don't know but I just want to be sure you have thought about it).

    Ok, maybe helpful...

    1. Split the resource into, say, three or more chunks and assemble at runtime.

    2. Make the resource either a RES or PBR file; do NOT use the inline resource definition statement, instead you will compile in the *.RES or *.PBR file. Note this is the way Pierre got this to work in Post #2 (Note also: Under PB now all resources have to be in the same format (RES or PBR), but we all know how to do that anyway).

    3. Use a third-party tool to compile a *.RES and then use a third-party linker to link that *.RES into your EXE.

    MCM

    Leave a comment:


  • Stuart McLachlan
    replied
    Tried twice with a 1GB file and gave up after PB sat using 25% CPU resources for over half an hour each time.

    No error though.

    Leave a comment:

Working...
X