When you sculpt terrain using the editor, it caches large amounts of "undo" information. The default is 50 undo's. You can see this by holding down on the left mouse button while taking a large swipe of the terrain using the flatten tool. You will see tens, if not hundreds of megabytes, of ram being eaten during each terrain edit. The longer you hold the LMB down and move the cursor, the more ram is eaten per undo. 200MB x 50 undo's = 10GB of ram, you can see that most computers could not fit this into ram. Taking smaller swipes reduces the amount of memory required for 1 "undo." 50MB x 50 undo's = 2.5GB, which is much more manageable for most systems. The other option is to lower the number of "undo's" the system will store. If large brush strokes cannot be avoided, try lowering the number of undo's to compensate, 20 should help.
TL:DR: Take short terrain editing strokes and lower the number of "undo's" the system stores.
