To stress test Space4J's cluster implementation, let's place a master node and a slave node under stress doing many reads and updates. Each node will have a predefined number of threads doing insert, delete, search and iteration at the same time in the Java collection in memory. A snapshot will also be taken from time to time by the slave node.

You can download the PhoneBookClusterStress.java program by clicking here. The source code is also listed below. (Note that to compile and run this problem you must include the space4.jar in the classpath)

To run this example you need to open two shell/dos windows, one for the master node and one for the slave node. You can also run each node in a separate machine over the network.

The arguments you must pass to the each program are:

-master or -slave = to indicate if this node is the master or the slave
IP address of the master
number of insert threads
number of delete threads
number of select threads
number of search threads
thread sleep time = how long each thread will sleep after each action
collection initial size = how many elements should the collection has before the threads start
snapshot time = how often should we take a snapshot

After executing the programs for some time, hit CTRL+C to print the statistics. Here is an example:

Master Program:
java -cp .;space4j.jar org.space4j.examples.phonebook.PhoneBookClusterStress -master 127.0.0.1 5 4 10 5 50 2000 1000 ... ... ... Selected 302 records! Found 14 recods! Selected 302 records! Selected 302 records! Selected 302 records! Selected 302 records! Selected 302 records! Found 24 recods! Found 21 recods! Found 23 recods! Selected 302 records! Selected 302 records! Inserted record number 5004 Inserted record number 5005 Deleted record number 2489 Inserted record number 5006 Deleted record number 4974 Deleted record number 655 Deleted record number 4926 Inserted record number 5007 ========== Results: Inserted = 2487 Deleted = 2041 Selected = 2334460 Searched = 86698 Snapshots= 0 Time: 39265 ms
Slave Program:
java -cp .;space4j.jar org.space4j.examples.phonebook.PhoneBookClusterStress -slave 127.0.0.1 5 4 10 5 50 2000 1000 ... ... Inserted record number 13273 Deleted record number 3267 Deleted record number 6578 Found 31 recods! Selected 302 records! Selected 302 records! Selected 302 records! Found 28 recods! Selected 302 records! Selected 302 records! Found 41 recods! Selected 302 records! Found 24 recods! Selected 302 records! Selected 302 records! Selected 302 records! Found 29 recods! Selected 302 records! Executing snapshot... =============================> Snapshot Complete! ========== Results: Inserted = 30 Deleted = 32 Selected = 45300 Searched = 2558 Snapshots= 3 Time: 781 ms
The source code of the PhoneBookClusterStress.java program:
package org.space4j.examples.phonebook; import java.util.Collection; import java.util.Iterator; import java.util.Random; import org.space4j.Space4J; public class PhoneBookClusterStress extends PhoneBookCluster { private final Thread[] insertThread; private final Thread[] deleteThread; private final Thread[] selectThread; private final Thread[] searchThread; private Thread snapshotThread; private Random rand = new Random(); private volatile long totalInserted = 0; private volatile long totalDeleted = 0; private volatile long totalSelected = 0; private volatile long totalSearched = 0; private volatile long snapshotsTaken = 0; private long started; private volatile boolean running = true; public PhoneBookClusterStress(Space4J space4j, final int nInsert, final int nDelete, final int nSelect, final int nSearch, final int sleepTime, final int tableSize, final int snapshotTime) throws Exception { super(space4j); // populate the table if needed... int size = users.size(); if (size < tableSize) { int diff = tableSize - size; for (int i = 0; i < diff; i++) { insertRandom(); System.out.print("\rInserting " + i + "... "); } } System.out.println(); insertThread = new Thread[nInsert]; deleteThread = new Thread[nDelete]; selectThread = new Thread[nSelect]; searchThread = new Thread[nSearch]; for (int i = 0; i < nInsert; i++) { insertThread[i] = new Thread(new Runnable() { public void run() { while (running) { try { Thread.sleep(sleepTime); handleInsert(); } catch (Exception e) { return; } } } }); } for (int i = 0; i < nDelete; i++) { deleteThread[i] = new Thread(new Runnable() { public void run() { while (running) { try { Thread.sleep(sleepTime); handleDelete(); } catch (Exception e) { return; } } } }); } for (int i = 0; i < nSelect; i++) { selectThread[i] = new Thread(new Runnable() { public void run() { while (running) { try { Thread.sleep(sleepTime); handleSelect(); } catch (Exception e) { return; } } } }); } for (int i = 0; i < nSearch; i++) { searchThread[i] = new Thread(new Runnable() { public void run() { while (running) { try { Thread.sleep(sleepTime); handleSearch(); } catch (Exception e) { return; } } } }); } if (snapshotTime > 0) { snapshotThread = new Thread( new Runnable() { public void run() { while (running) { try { Thread.sleep(snapshotTime); System.out.println("Executing snapshot..."); PhoneBookClusterStress.this.executeSnapshot(); System.out.println( "=============================> Snapshot Complete!"); snapshotsTaken++; } catch (Exception e) { return; } } } }); } } public void start() { for (int i = 0; i < insertThread.length; i++) { insertThread[i].start(); } for (int i = 0; i < deleteThread.length; i++) { deleteThread[i].start(); } for (int i = 0; i < selectThread.length; i++) { selectThread[i].start(); } for (int i = 0; i < searchThread.length; i++) { searchThread[i].start(); } snapshotThread.start(); } protected int insertRandom() throws Exception { // insert a random phone number... StringBuilder sb = new StringBuilder(); sb.append("Oliveira").append(rand.nextInt(100) + 1); return this.addNumber("Sergio", sb.toString(), "123-4321", rand.nextInt(100) + 1); } protected int deleteRandom() throws Exception { // delete a random phone number // (guess a number, iterate to choose a phone and delete it) Iterator iter = users.keySet().iterator(); int count = rand.nextInt(users.size() / 10) + 1; int index = 0; int id = -1; while (iter.hasNext() && index++ <= count) { id = (Integer) iter.next(); } if (id > 0) { this.delUser(id); } return id; } private void handleInsert() throws Exception { int id = insertRandom(); System.out.println("Inserted record number " + id); totalInserted++; } private void handleDelete() throws Exception { int id = deleteRandom(); if (id == -1) { System.out.println("Could not delete anything!"); } else { System.out.println("Deleted record number " + id); totalDeleted++; } } private void handleSelect() { // select any 300 phone numbers and loop through them... Iterator iter = users.values().iterator(); int count = 0; while (iter.hasNext() && count++ <= 300) { iter.next(); } System.out.println("Selected " + count + " records!"); totalSelected += count; } private void handleSearch() { // choose a user and try to find his phone number... StringBuilder sb= new StringBuilder(); sb.append("Oliveira").append(rand.nextInt(100) + 1); Collection coll = this.findUsers(sb.toString()); int count = 0; Iterator iter = coll.iterator(); while (iter.hasNext()) { count++; iter.next(); } System.out.println("Found " + count + " recods!"); totalSearched += count; } private void interruptAll() { for(int i=0;i<insertThread.length;i++) insertThread[i].interrupt(); for(int i=0;i<deleteThread.length;i++) deleteThread[i].interrupt(); for(int i=0;i<selectThread.length;i++) selectThread[i].interrupt(); for(int i=0;i<searchThread.length;i++) searchThread[i].interrupt(); } public static void main(String[] args) throws Exception { if (args.length >= 3) { if (args[2].equals("-nostress")) { Space4J space4j = createSpace4J(args[0], args[1]); PhoneBookCluster phone = new PhoneBookCluster(space4j); run(phone); return; } } if (args.length != 9) { System.out.println( "format: java PhoneBookClusterStress [-master|-slave] <ip> [-nostress] <number of insert threads> <number of delete threads> " + "<number of select threads> <number of search threads> <thread sleep time> <table size> <snapshot time>"); return; } int nInsert, nDelete, nSelect, nSearch, sleepTime, tableSize, snapTime; nInsert = Integer.parseInt(args[2]); nDelete = Integer.parseInt(args[3]); nSelect = Integer.parseInt(args[4]); nSearch = Integer.parseInt(args[5]); sleepTime = Integer.parseInt(args[6]); tableSize = Integer.parseInt(args[7]); snapTime = Integer.parseInt(args[8]); // let the slaves take the snapshot if (args[0].equals("-master")) { snapTime = 0; } Space4J space4j = createSpace4J(args[0], args[1]); final PhoneBookClusterStress stress = new PhoneBookClusterStress(space4j, nInsert, nDelete, nSelect, nSearch, sleepTime, tableSize, snapTime); stress.started = System.currentTimeMillis(); Runtime.getRuntime().addShutdownHook(new Thread() { @Override public void run() { long totalInserted = stress.totalInserted; long totalDeleted = stress.totalDeleted; long totalSelected = stress.totalSelected; long totalSearched = stress.totalSearched; long snapshotsTaken = stress.snapshotsTaken; long now = System.currentTimeMillis(); stress.running = false; stress.interruptAll(); try { Thread.sleep(1500); } catch(Exception e) { } System.out.println("\n\n========== Results:"); System.out.println("Inserted = " + totalInserted); System.out.println("Deleted = " + totalDeleted); System.out.println("Selected = " + totalSelected); System.out.println("Searched = " + totalSearched); System.out.println("Snashopts= " + snapshotsTaken); System.out.println("Time: " + (now - stress.started) + " ms"); } }); stress.start(); } }

This message was edited 7 times. Last update was at 05/09/2008 16:24:04