memory use in large data-structures manipulation/processing

I have a number of large (~100 Mb) files which I'm regularly processing. While I'm trying to delete unneeded data structures during processing, memory consumption is a bit too high. so, I was wondering is there a way to 'efficiently' manipulate large data, e.g.:

def read(self, filename):
    fc = read_100_mb_file(filename)
    self.process(fc)
def process(self, content):
    # do some processing of file content

is there a duplication of data structures? isn't it more memory efficient to use class-wide variable like self.fc?

how to garbage-collect? I know about gc module, but do I call it after i del fc for example? does garbage collector called after a del statement at all? when should I use garbage collection?

update
p.s. 100 Mb is not a problem in itself. but float conversion, further processing add significantly more to both working set and virtual size (i'm on windows).

From stackoverflow

• Before you start tearing your hair out over the garbage collector, you might be able to avoid that 100mb hit of loading the entire file into memory by using a memory-mapped file object. See the mmap module.

  SilentGhost : 100 Mb is just fine, problem starts when it hits 1.7 Gb of virtual memory

  Crashworks : Yikes! That sounds more like you are hanging onto references to many things, so that the garbage collector can't clean them up. This can happen if you save off a reference to your intermediate data in the processing class.

  SilentGhost : that's exactly my question!

• Don't read the entire 100 meg file in at a time. Use streams to process a little bit at a time. Check out this blog post that talks about handling large csv and xml files. http://lethain.com/entry/2009/jan/22/handling-very-large-csv-and-xml-files-in-python/

  Here is a sample of the code from the article.

  from __future__ import with_statement # for python 2.5
  
  with open('data.in','r') as fin:
      with open('data.out','w') as fout:
          for line in fin:
              fout.write(','.join(line.split(' ')))
  

  SilentGhost : it doesn't seem to scale in terms of code, I don't need just to rearrange bits, there's more processing involved

  Sam Corder : Once you have parsed a detail line and done your reduction calculations make sure you aren't hanging on to any of the objects created from parsing the details. Python GC is reference based. As long as there is a reference to an object it won't get GC'ed.

  Sam Corder : Just to add. If you have two objects that refer to each other they will never be garbage collected unless one of them lets go of the reference to the other. Check for this kind of circular reference if you see your memory usage ballooning and thing you the objects should be out of scope.

  Torsten Marek : @Sam Corder: Cyclic garbage collection has long since been added to Python.

  Sam Corder : @Torsten Marek: Very cool. Thanks for the correction.

• So, from your comments I assume that your file looks something like this:

  item1,item2,item3,item4,item5,item6,item7,...,itemn
  

  which you all reduce to a single value by repeated application of some combination function. As a solution, only read a single value at a time:

  def read_values(f):
      buf = []
      while True:
          c = f.read(1)
          if c == ",":
              yield parse("".join(buf))
              buf = []
          elif c == "":
              yield parse("".join(buf))
              return
          else:
              buf.append(c)
  
  with open("some_file", "r") as f:
       agg = initial
       for v in read_values(f):
           agg = combine(agg, v)
  

  This way, memory consumption stays constant, unless agg grows in time.

  1. Provide appropriate implementations of initial, parse and combine
  2. Don't read the file byte-by-byte, but read in a fixed buffer, parse from the buffer and read more as you need it

  3. This is basically what the builtin reduce function does, but I've used an explicit for loop here for clarity. Here's the same thing using reduce:

     with open("some_file", "r") as f:
         agg = reduce(combine, read_values(f), initial)
     

  I hope I interpreted your problem correctly.

  SilentGhost : i'm sorry if I've put it clumsy, but by reduce I meant "make 32 Kb from 100 Mb"

  Torsten Marek : No, I didn't mean that, I meant the reduce builtin.

  J.F. Sebastian : I've added `reduce` example.

  J.F. Sebastian : btw, `f.read()` should be `f.read(1)` in your code. And open("somefile", r) -> open("somefile", "r").

  Torsten Marek : @J.F.: Ah, the joys of coding without testing. I've actually tried out the code and used f.read(1) there.

  S.Lott : +1: Process incrementally

• First of all, don't touch the garbage collector. That's not the problem, nor the solution.

  It sounds like the real problem you're having is not with the file reading at all, but with the data structures that you're allocating as you process the files. Condering using del to remove structures that you no longer need during processing. Also, you might consider using marshal to dump some of the processed data to disk while you work through the next 100mb of input files.

  For file reading, you have basically two options: unix-style files as streams, or memory mapped files. For streams-based files, the default python file object is already buffered, so the simplest code is also probably the most efficient:

    with open("filename", "r") as f:
      for line in f:
         # do something with a line of the files
  

  Alternately, you can use f.read([size]) to read blocks of the file. However, usually you do this to gain CPU performance, by multithreading the processing part of your script, so that you can read and process at the same time. But it doesn't help with memory usage; in fact, it uses more memory.

  The other option is mmap, which looks like this:

    with open("filename", "r+") as f:
      map = mmap.mmap(f.fileno(), 0)
      line = map.readline()
      while line != '':
         # process a line
         line = map.readline()
  

  This sometimes outperforms streams, but it also won't improve memory usage.

• I'd suggest looking at the presentation by David Beazley on using generators in Python. This technique allows you to handle a lot of data, and do complex processing, quickly and without blowing up your memory use. IMO, the trick isn't holding a huge amount of data in memory as efficiently as possible; the trick is avoiding loading a huge amount of data into memory at the same time.

  Van Gale : Gah, as soon as I saw the question I jumped in to answer with link to the Beazley stuff and saw you give it as an answer already. Oh well, have to vote you up +1 instead! Just wish I could give it more than +1.

• In your example code, data is being stored in the fc variable. If you don't keep a reference to fc around, your entire file contents will be removed from memory when the read method ends.

  If they are not, then you are keeping a reference somewhere. Maybe the reference is being created in read_100_mb_file, maybe in process. If there is no reference, CPython implementation will deallocate it almost immediatelly.

  There are some tools to help you find where this reference is, guppy, dowser, pysizer...