Messages posted by: saoj

You get a OutOfMemoryError and you have to increase your heap size with the -Xms options or eventually your physical RAM. Google about Java Heap size command options.

Amount of RAM should not be a problem, unless you have a really huge database. In this case you should probably use a regular relational database.

-Sergio

Are all serializable objects being held in memory during the working with Space4j????

Yes, all objects are in memory, but they are no serialized in memory. They get serialized when they are logged to disk in a command or when a snapshot is taken.

I mean, let's suppose that I've added 1K of objects into space4j...My question is, are all of these objects in memory? Or does it mean that half of these objects (the most used) is in memory and the second half is saved on HD???

All objects are always in memory, even if they are stale objects that are not being accessed.

I thought about a *passivation* strategy for Space4J in the past, that would swap unused/old objects from memory to disk. But that complicates the whole thing and loses the focus of Space4J which is in-memory access straight through collections.

If you have a logging table that grows indefintely and the information is not accessed regularly, you will be better off with a text log or a relational database. To keep this information in memory would be just a waste of RAM.

1) All objects all serialized in the "./space4_db" directory. Is somehow possible to set the directory and file where the objects will be serialized?

Download the beta jar from http://www.space4j.org/beta/space4j.jar and use the SimpleLogger.setDir("c:\\mydb") for example. You can also use a relative path like SimpleLogger.setDir("mydb").

2) I want to use REGULAR index (unique-index, non-sorted), but I didn't find an example of how to get my stored value....Of course I checked an example on this forum

User u = usersById.get(2345);

but I don't understand one thing, what type of the object is usersById???? Can you provide full example where REGULAR index is used.

usersById in this example is the map which is returned by the Index objetct.

Once you have created the index object by calling im.createIndex(indx, space4j); you can get its map to perform the lookup:

Note that you need a key object. You can even have composite indexes with composite keys (more than one attribute).

Also note that is recommended that you store your objects in the space inside Maps not Lists. The reason for that is the same reason for always having a Primary Key in a database table.

I have been thinking about this and my bet is that the bottleneck is on disk I/O.

Disk I/O after a write (or a commit in relational DB) is pretty much unescapable, because you need to make sure data is persisted and safe in case of a crash.

I am wondering who is faster. A relational database doing multiple inserts and commits in sequence or Space4J.

If you are really concerned about this speed, you can perform the disk I/O asynchronously, but the price is clear: you will not be sure when your write is persisted and safe in disk. After a crash, you will end up losing some past sucessful writes.

So two questions remain:

1) Is Space4J as fast as any regular relational database when it comes to muliple and consective inserts/commits ?

2) Is there a scenario where doing async disk i/o is actually desirable considering its cost?

Let me know your thoughts...

Hi Fred!

I would not expect this slowliness. Now I am curious what is the bottleneck: I/O?

You would have to take that Space4J source code and benchmark the writing process, step by step, to find out where the bottleneck is.

-Sergio

xolotl wrote:
By the way, the fuss over simplistic designs like Prevayler or Space4j tends to obcure
the fact that there are fully-fledged transactional db engines designed to operate
in-memory although not in-process, an example that come to mind being TimesTen
(now an Oracle property). So you can have your cake an eat it, too.

I agree. It is not about speed anymore, I would think. It is about a simple way to perform reads straight from collections.

SQL, JDBC Driver, ORM, connection pooling may be too much for many apps.

Prevayler had a not so good attitude that it was the salvation of the world and that databases were evil.

This was a joke in my personal opinion. Database are awesome!

Space4J is just another option...

xolotl wrote:Can I modify my data while someone else is reading it? Yes! Does that mean that readers might see the ground shift under them, e.g., while they are iterating over a collection?

Good question. I am not an expert on java.util.concurrent collections. We would have to ask this question to Doug Lee. However I imagine that if you are iterating (reading) and someone else is modifying, the results are king of unpredictable. But you will NOT get a ConcurrentModificationException.

I wonder how relational database handles that situation... This topic is not clear to me, but I am willing to learn from the experts... It has to do with isolation level.

1. What do you mean? An exception when executing a command will not log this command. Can you explain better?

2. Any help implementing a generics layer will be highly appreciated. Generics is good stuff and Space4J should take advantage of it. Let me know if you want to commit something to the project about that.

3. Not that I know of. But the stress test should be a good indication that it is reliable under heavy load. Of course it can have hidden bugs, but a bug does not exist until it introduces itself in a real-case scenario.

agile.guo wrote:
In my application,there will be many indexes to be created,and some indexes should be created dynamically,is there any question?

No problem at all. Indexes can be created dynamic. Just like with any relational database, the indexation process can take long if your dataset is large. I would not worry for anything < 500k.

If you are indexing a huge dataset, it is better to do it offline, with your system off, to avoid write locks. Same problem with any relational database.

And how should I can calculate the space of indexes?

Not easily. Java will not easily tell you the ammount of memory an object is taking. There are hacks to do this. Google it and if you get a good solution please share with me.

findUsers iterates through all the found items before it returns the result, which is in turn iterated again.

See the problem?

A better approach instead of iterating through all elements and sticking them in a new list object would be to create a custom iterator so that the findUsers method would return this iterator.

This is not a problem for small results, but for big results (>100k) this is not a good idea.

Did you get it?

To summarize: findUsers return a collection that we just iterate through. Returning a iterator would be more appropriate for large subsets.

But, if you assume that you want a Collection with the returned elements, then this implemenation is appropriate.

Sorry for the late response.

Yes you can use Space4J with your license. No problem...

See the PhoneBook example to have an idea how you can easily persist objects with Space4J. Then decide if you want to follow that approach instead of the standard relational database approach.

-Sergio

We know that it is impossible to abandon your good and old relational database. You will probably need to do offline reports and heavy queries and a relational database is perfect for that. So what you need is to replicate all your Space4J commands to a relational database. To do this you can implement the executeSQL method for your commands so that you can later read the logs and execute them against a relational database.

Check the method below from the Command interface:

ToDo: Create a standalone program that read the logs and execute them against a JDBC connection.

Clustering can be useful for load-balance and fault-tolerance. It also allows a snapshot to be taken without placing the whole system in read-only mode. That's because you can use a slave to take snapshot while the rest of the nodes continue to run like nothing as happened, in other words, only the slave taking the snapshot needs to enter read-only mode.

To create a Space4J cluster is very easy and simple. Refer to the PhoneBookCluster.java program for an example. We also have a stress program to test the cluster feature.

The source code for PhoneBookCluster.java is below:

package org.space4j.examples.phonebook;

import org.space4j.Space4J;
import org.space4j.implementation.MasterSpace4J;
import org.space4j.implementation.SlaveSpace4J;

/**
 * This is to show you how simple it is to make a Space4J cluster!
 * 
 * To make a cluster, open up two shells (or DOS) and execute in the first:
 * 
 * java org.space4j.demos.phonebook.PhoneBookCluster master
 * 
 * and in the second:
 * 
 * java org.space4j.demos.PhoneBookCluster slave 127.0.0.1
 * 
 * You can also use two different machines. Just pass the master IP address to the slave instead of 127.0.0.1.
 * IMPORTANT: If you use two different machines, you must have the main Space4J dir (space4j_db) mounted, so 
 * both machines can have access to it.
 */
public class PhoneBookCluster extends PhoneBookIndexing {
    
    public PhoneBookCluster(Space4J space4j) throws Exception {
    	
        super(space4j);
    	
    }
    
    static Space4J createSpace4J(String master, String master_ip) throws Exception {
    	
        Space4J space4j = null;
    	
        if (master.equals("-master")) {
        	
            space4j = new MasterSpace4J("PhoneBook");
        	
        } else if (master.equals("-slave")) {
        	
            if (master_ip != null) {
            	
                space4j = new SlaveSpace4J("PhoneBook", master_ip);
            	
            } else {
            	
                space4j = new SlaveSpace4J("PhoneBook", "127.0.0.1");
            	
            }
        }
        
        return space4j;
    }
    
    public static void main(String[] args) throws Exception {
    	
        if (args.length < 1 || args.length > 2) {
    		
            System.out.println(
                    "format: java PhoneBookCluster [-master|-slave] <IP def: 127.0.0.1>");
    		
            return;
        }
    	
        Space4J space4j = createSpace4J(args[0],
                args.length > 1 ? args[1] : null);
    	
        PhoneBookCluster book = new PhoneBookCluster(space4j);
        
        run(book);
    }
}

When your collection becomes too big, even in-memory searches can become expensive. Imagine a Java collection with 1 million users and you need to find all users with the last name equals to "Smith". Iterating through all elements is clearly not an option. What you need is INDEXES.

Space4J supports 4 different types of indexes through maps, in other words, when you create an index a new Java Map is created to store the index.

The four types of index are:

Index.TYPE.REGULAR = Unique index non-sorted. This is used to get an user by its id for example, where each user has an unique id.

Index.TYPE.SORTED = Unique index sorted. This is used to get a list of users in a range of ids (ex: 4000 < id < 5000). Also used when you need to fetch users sorted by the indexed field.

Index.TYPE.MULTI = Non-unique index non-sorted. This is used to get all users with age equal to 23.

Index.TYPE.MULT_SORTED = Non-unique index sorted. This is used to get all users with age >= 23 and age <= 40. Also used when you need to fech users sorted by the indexed field.

The PhoneBookIndexing.java program provides examples of how to create and user indexes for your collections in memory. Below is the source code:

package org.space4j.examples.phonebook;

import java.util.Collection;

import org.space4j.Space4J;
import org.space4j.implementation.SimpleSpace4J;
import org.space4j.indexing.Index;
import org.space4j.indexing.IndexManager;
import org.space4j.indexing.Key;
import org.space4j.indexing.MultiMap;
import org.space4j.indexing.MultiSortedMap;

public class PhoneBookIndexing extends PhoneBook {
	
    private static final String INDX_LASTNAME = "indx_lastname";
    
    private static final String INDX_AGE = "indx_age";
	
    private final Index<User> indexByLastName;
    
    private final Index<User> indexByAge;
	
    public PhoneBookIndexing(Space4J space4j) throws Exception {
    	
        super(space4j);
    	
        IndexManager im = space.getIndexManager();
    	
        // If index does not exist, let's create it...
        if (!im.checkIndex(INDX_LASTNAME)) {
            
            /*
             * Index name = INDX_LASTNAME
             * Map in the Space that will be indexed = MAP_NAME
             * Type of index = MULTI = non-unique index
             * Object that will be indexed = User.class
             * Atributes of the object that will be indexed = "lastName"
             * 
             * We want to create this index because we will be searching
             * for Users by lastName. We don't want to do a full table
             * scan anymore like in the example without indexes.
             */
        	
            Index<User> indx = new Index<User>(INDX_LASTNAME, MAP_NAME, Index.TYPE.MULTI, User.class, "lastName");
        	
            boolean ok = im.createIndex(indx, space4j);
            
            if (ok) {
            	
                System.out.println(indx + " created!");
            	
            } else {
            	
                System.out.println(indx + " could not be created!");
            }
        }
        
        // If index does not exist, let's create it...
        if (!im.checkIndex(INDX_AGE)) {
            
            /*
             * Index name = INDX_AGE
             * Map in the Space that will be indexed = MAP_NAME
             * Type of index = MULTI = non-unique sorted index
             * Object that will be indexed = User.class
             * Atributes of the object that will be indexed = "age"
             * 
             * We want to create this index because we will be searching
             * for Users by age. The index will be SORTED because we want
             * to search for users in a age range. (age > min and age < max)
             */
         
            Index<User> indx = new Index<User>(INDX_AGE, MAP_NAME, Index.TYPE.MULTI_SORTED, User.class, "age");
         
            boolean ok = im.createIndex(indx, space4j);
            
            if (ok) {
               
                System.out.println(indx + " created!");
               
            } else {
               
                System.out.println(indx + " could not be created!");
            }
        }

indexByLastName = im.getIndex(INDX_LASTNAME);
        indexByAge = im.getIndex(INDX_AGE);
    }
    
    /*
     * Let's override our search method to use our
     * index instead of doing a full table scan.
     */
    @Override
    public Collection<User> findUsers(String last) {
	
    	// Remember that our index is of the type MULTI.
    	
        MultiMap<User> map = indexByLastName.getMap();
        
        // A multi-map will store a Set for each key.
        // That's why our index is NON-UNIQUE meaning the same
        // key (lastName) can be associated with different entries in the map.
        // That makes sense as some users can have the same last name.
    	
       return map.get(last);
    }
    
    @Override
    public Collection<User> findUsers(int minAge, int maxAge) {
       
       // Remember that our index is of the type MULTI_SORTED.
       
       MultiSortedMap<User> map = indexByAge.getMap();
       
       // A multi-sorted-map will store a Set for each key.
       // That's why our index is NON-UNIQUE meaning the same
       // key (minAge, maxAge) can be associated with different entries in the map.
       // The map is also sorted by its key, meaning we can
       // perform range searches efficiently like below.
       
       return map.subMapValues(new Key(minAge), true, new Key(maxAge), true);
    }
    
    public static void main(String[] args) throws Exception {
    	
        PhoneBookIndexing book = new PhoneBookIndexing(
                new SimpleSpace4J("PhoneBook"));

run(book);
    	
    }
}

To stress test Space4J's cluster implementation, let's place a master node and a slave node under stress doing many reads and updates. Each node will have a predefined number of threads doing insert, delete, search and iteration at the same time in the Java collection in memory. A snapshot will also be taken from time to time by the slave node.

You can download the PhoneBookClusterStress.java program by clicking here. The source code is also listed below. (Note that to compile and run this problem you must include the space4.jar in the classpath)

To run this example you need to open two shell/dos windows, one for the master node and one for the slave node. You can also run each node in a separate machine over the network.

The arguments you must pass to the each program are:

-master or -slave = to indicate if this node is the master or the slave

IP address of the master

number of insert threads

number of delete threads

number of select threads

number of search threads

thread sleep time = how long each thread will sleep after each action

collection initial size = how many elements should the collection has before the threads start

snapshot time = how often should we take a snapshot

After executing the programs for some time, hit CTRL+C to print the statistics. Here is an example:

Master Program:

Slave Program:

The source code of the PhoneBookClusterStress.java program:

package org.space4j.examples.phonebook;

import java.util.Collection;
import java.util.Iterator;
import java.util.Random;

import org.space4j.Space4J;

public class PhoneBookClusterStress extends PhoneBookCluster {
	
    private final Thread[] insertThread;
	
    private final Thread[] deleteThread;
	
    private final Thread[] selectThread;
	
    private final Thread[] searchThread;
	
    private Thread snapshotThread;
    
    private Random rand = new Random();
    
    private volatile long totalInserted = 0;
    private volatile long totalDeleted = 0;
    private volatile long totalSelected = 0;
    private volatile long totalSearched = 0;
    private volatile long snapshotsTaken = 0;
    
    private long started;
    
    private volatile boolean running = true;
	
    public PhoneBookClusterStress(Space4J space4j, final int nInsert, final int nDelete, final int nSelect, final int nSearch,
            final int sleepTime, final int tableSize, final int snapshotTime) throws Exception {
		
        super(space4j);
		
        // populate the table if needed...
		
        int size = users.size();
		
        if (size < tableSize) {
			
            int diff = tableSize - size;
			
            for (int i = 0; i < diff; i++) {
				
                insertRandom();
				
                System.out.print("\rInserting " + i + "...                 ");
            }
        }
		
        System.out.println();
		
        insertThread = new Thread[nInsert];
		
        deleteThread = new Thread[nDelete];
		
        selectThread = new Thread[nSelect];
		
        searchThread = new Thread[nSearch];
		                         
        for (int i = 0; i < nInsert; i++) {
            insertThread[i] = new Thread(new Runnable() {
			
                public void run() {
				
                    while (running) {
					
                        try {
						
                            Thread.sleep(sleepTime);
						
                            handleInsert();
						
                        } catch (Exception e) {
						
                            return;
                        }
                    }
                }
			
            });
        }
		
        for (int i = 0; i < nDelete; i++) {
            deleteThread[i] = new Thread(new Runnable() {
			
                public void run() {
				
                    while (running) {
					
                        try {
						
                            Thread.sleep(sleepTime);
						
                            handleDelete();
						
                        } catch (Exception e) {
						
                            return;
                        }
                    }
                }
			
            });
        }
		
        for (int i = 0; i < nSelect; i++) {
            selectThread[i] = new Thread(new Runnable() {
			
                public void run() {
				
                    while (running) {
					
                        try {
						
                            Thread.sleep(sleepTime);
						
                            handleSelect();
						
                        } catch (Exception e) {
						
                            return;
                        }
                    }
                }
			
            });
        }
		
        for (int i = 0; i < nSearch; i++) {
            searchThread[i] = new Thread(new Runnable() {
			
                public void run() {
				
                    while (running) {
					
                        try {
						
                            Thread.sleep(sleepTime);
						
                            handleSearch();
						
                        } catch (Exception e) {
						
                            return;
                        }
                    }
                }
			
            });
        }
        
        if (snapshotTime > 0) {
		
	        snapshotThread = new Thread(
	                new Runnable() {
				
	            public void run() {
					
	                while (running) {
						
	                    try {
							
	                        Thread.sleep(snapshotTime);
							
	                        System.out.println("Executing snapshot...");
							
	                        PhoneBookClusterStress.this.executeSnapshot();
							
	                        System.out.println(
	                                "=============================> Snapshot Complete!");
	                        
	                        snapshotsTaken++;
							
	                    } catch (Exception e) {
							
	                        return;
	                    }
	                }
	            }
				
	        });
        }
    }
	
    public void start() {
		
        for (int i = 0; i < insertThread.length; i++) {
            insertThread[i].start();
        }
		
        for (int i = 0; i < deleteThread.length; i++) {
            deleteThread[i].start();
        }
		
        for (int i = 0; i < selectThread.length; i++) {
            selectThread[i].start();
        }
		
        for (int i = 0; i < searchThread.length; i++) {
            searchThread[i].start();
        }
		
        snapshotThread.start();
    }
	
    protected int insertRandom() throws Exception {
    	
    	// insert a random phone number...
		
        StringBuilder sb = new StringBuilder();
        
        sb.append("Oliveira").append(rand.nextInt(100) + 1);
		
        return this.addNumber("Sergio", sb.toString(), "123-4321", rand.nextInt(100) + 1);
    }
	
    protected int deleteRandom() throws Exception {
    	
    	// delete a random phone number
    	// (guess a number, iterate to choose a phone and delete it)

Iterator iter = users.keySet().iterator();
		
        int count = rand.nextInt(users.size() / 10) + 1;
		
        int index = 0;
		
        int id = -1;
		
        while (iter.hasNext() && index++ <= count) {
			
            id = (Integer) iter.next();
			
        }
		
        if (id > 0) {
			
            this.delUser(id);
			
        }
		
        return id;
    }
	
    private void handleInsert() throws Exception {
		
        int id = insertRandom();
		
        System.out.println("Inserted record number " + id);
        
        totalInserted++;
    }
	
    private void handleDelete() throws Exception {
		
        int id = deleteRandom();
		
        if (id == -1) {
			
            System.out.println("Could not delete anything!");
			
        } else {
			
            System.out.println("Deleted record number " + id);
            
            totalDeleted++;
        }
    }
	
    private void handleSelect() {
    	
    	// select any 300 phone numbers and loop through them...
		
        Iterator iter = users.values().iterator();
		
        int count = 0;
		
        while (iter.hasNext() && count++ <= 300) {
			
            iter.next();
        }
		
        System.out.println("Selected " + count + " records!");
        
        totalSelected += count;
    }

private void handleSearch() {
    	
    	// choose a user and try to find his phone number...
    	
    	StringBuilder sb= new StringBuilder();
		
        sb.append("Oliveira").append(rand.nextInt(100) + 1);
		
        Collection coll = this.findUsers(sb.toString());
		
        int count = 0;
		
        Iterator iter = coll.iterator();
		
        while (iter.hasNext()) {
			
            count++;
			
            iter.next();
        }
		
        System.out.println("Found " + count + " recods!");
        
        totalSearched += count;
		
    }
    
    private void interruptAll() {
    	
    	for(int i=0;i<insertThread.length;i++) insertThread[i].interrupt();
    	
    	for(int i=0;i<deleteThread.length;i++) deleteThread[i].interrupt();
    	
    	for(int i=0;i<selectThread.length;i++) selectThread[i].interrupt();
    	
    	for(int i=0;i<searchThread.length;i++) searchThread[i].interrupt();
    }
	
    public static void main(String[] args) throws Exception {
		
        if (args.length >= 3) {
			
            if (args[2].equals("-nostress")) {
				
                Space4J space4j = createSpace4J(args[0], args[1]);
					
                PhoneBookCluster phone = new PhoneBookCluster(space4j);
				
                run(phone);
				
                return;
            }
			
        }
		
        if (args.length != 9) {
			
            System.out.println(
                    "format: java PhoneBookClusterStress [-master|-slave] <ip> [-nostress] <number of insert threads> <number of delete threads> "
                            + "<number of select threads> <number of search threads> <thread sleep time> <table size> <snapshot time>");
			
            return;
        }
		
        int nInsert, nDelete, nSelect, nSearch, sleepTime, tableSize, snapTime;
		
        nInsert = Integer.parseInt(args[2]);
		
        nDelete = Integer.parseInt(args[3]);
		
        nSelect = Integer.parseInt(args[4]);
		
        nSearch = Integer.parseInt(args[5]);
		
        sleepTime = Integer.parseInt(args[6]);
		
        tableSize = Integer.parseInt(args[7]);
		
        snapTime = Integer.parseInt(args[8]);
        
        // let the slaves take the snapshot
        if (args[0].equals("-master")) {
        	snapTime = 0;
        }
		
        Space4J space4j = createSpace4J(args[0], args[1]);
		
        final PhoneBookClusterStress stress = new PhoneBookClusterStress(space4j,
                nInsert, nDelete, nSelect, nSearch, sleepTime, tableSize,
                snapTime);
        
        stress.started = System.currentTimeMillis();
        
        Runtime.getRuntime().addShutdownHook(new Thread() {

@Override
            public void run() {
            	
            	long totalInserted = stress.totalInserted;
            	long totalDeleted = stress.totalDeleted;
            	long totalSelected = stress.totalSelected;
            	long totalSearched = stress.totalSearched;
            	long snapshotsTaken = stress.snapshotsTaken;
            	long now = System.currentTimeMillis();
            	
            	stress.running = false;
            	
            	stress.interruptAll();
            	
            	try { Thread.sleep(1500); } catch(Exception e) { }
            	
            	System.out.println("\n\n========== Results:");
            	System.out.println("Inserted = " + totalInserted);
            	System.out.println("Deleted  = " + totalDeleted);
            	System.out.println("Selected = " + totalSelected);
            	System.out.println("Searched = " + totalSearched);
            	System.out.println("Snashopts= " + snapshotsTaken);
            	System.out.println("Time: " + (now - stress.started) + " ms");
            	
            }
        });
		
        stress.start();
    }
	
}